Ingestion mapping editor
Outcome
Per-feed mappings are authored, dry-run previewed, published, and rolled back without a developer in the loop.
Prerequisites
ingestion.createto author and publish.ingestion.readto dry-run.
Mental model
| Concept | Detail |
|---|---|
| A mapping is per source feed | Each row in ingestion.mapping_definition is one immutable version. The current one is is_current=TRUE. |
| Publishing creates a new row | Server takes a per-feed pg_advisory_xact_lock, computes version = MAX(version) + 1, flips the prior current row to is_current=FALSE, inserts the new row in a single transaction. Concurrency-safe. |
| Drafts are client-side only | No DRAFT state on the table — closing the browser tab abandons the draft. Collaborate by copying YAML out of band. |
| Rollback = republish prior YAML | No is_current=TRUE toggle. Open the prior version → "Edit as draft" → Publish. The new version's YAML matches the old; it lands as vN+1. |
Author a new mapping
Sign in as an operator with
ingestion.create.Open
/admin/ingestion/mappingsand pick the target feed in the combobox. The page navigates to/admin/ingestion/mappings/:feedId.Click New draft — lands on the editor with a starter YAML (
charge_eventtarget, singleexternal_idfield).Visual builder tab: add fields, pick the transform type, set parameters (e.g.
inputFormat=MM/DD/YYYYforDATE). Validation rules edit the same way.YAML tab: free-form Monaco edit. The schema validator (zod-mirrored from
MappingService.validateMappingYaml) flags missingtarget.type, emptyfields, unknown transform kinds, etc. inline.Diff tab: side-by-side Monaco diff against the currently published version. Empty when there is no prior version.
Dry run: paste a sample row (CSV header+row, or JSON object) into the right-hand panel. Each field's mapped value (and any transform-time error) renders in the preview table.
LOOKUPandCUSTOMtransforms show a "server-side only" notice — they require a real run.Click Publish. Pick
Effective from(defaults to today) and optionalEffective to. Confirm. The toast announces the newvN. The version list automatically picks up the new row and marks itCurrent.
Roll back to a prior version
Open
/admin/ingestion/mappings/:feedIdand click the prior version in the list.Click Edit as draft in the YAML pane header.
Confirm the YAML matches the desired prior content (no edits needed).
Click Publish. The new row gets
vN+1but the YAML matchesvN-1. The previously brokenvNisSuperseded.
Effective-date windows
effective_fromis required.effective_tois optional. Set it when sunsetting a vendor or switching to a new schema on a known date.getEffectiveMapping(feedId, asOfDate)resolves the row that is current AND whose effective window containsasOfDate. Multiple current rows with overlapping windows is not a defined state — the publish path always flips the prior current row, so this only matters when seeding via SQL.
Validation
| Check | Expected |
|---|---|
| Schema validator (YAML tab) | Green before Publish |
| Dry-run preview | Sample row maps as expected |
mapping_definition after publish | New row, is_current=true; prior row is_current=false |
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Publish 400 with mapping_yaml missing target.type | Client + server schema have drifted | Re-run the schema validator; if still failing, file a follow-up to re-sync apps/rcm-app/.../yaml-schema.ts with mapping-service.ts. |
Publish 409 UNIQUE violation on (source_feed_id, version) | Concurrent publishes serialized via pg_advisory_xact_lock; this should not happen | Inspect mapping_definition and re-run migration if version numbers are non-contiguous. |
| Editor opens with empty YAML for a feed that has versions | Route response missing rawYaml. Confirm feed isn't filtered by effective_from > asOfDate | If getEffectiveMapping returns null, editor falls back to default scaffold. |
Cross-references
- Ingestion feed admin for SFTP / push-API config.
- Fixed-width parser config for the parallel
parser_configeditor on FIXED_WIDTH feeds.