Handle an SFTP failure 🚨

Outcome

EDI transactions stuck in SUBMITTING status are unblocked, the underlying SFTP issue (auth, credential expiry, network) is identified and fixed, and failed transactions are retried.

Prerequisites

• PLATFORM_ADMIN or EDI_OPERATOR (rotation + retry rights).
• SSH client with sftp for direct connectivity probes.
• Access to Key Vault for credential inspection / rotation.

Symptoms

• EDI transactions stuck in SUBMITTING status for > 10 minutes.
• Error logs showing connection timeouts or authentication failures.
• Response poller (271 / 277CA / 835) finding no new files for hours.

Decision flow

Steps

1. Diagnose the affected transactions:

   SELECT edi_transaction_id, direction, tx_type, status,
          error_details, retry_count, created_at
   FROM billing.edi_transaction
   WHERE status IN ('ERROR', 'SUBMITTING')
     AND created_at > NOW() - INTERVAL '24 hours'
   ORDER BY created_at DESC;

   Note the trading_partner and the count — is this one partner or all of them? One partner narrows to credential/connectivity; all partners points at network or app-side.

2. Probe SFTP connectivity from the rcm-core host:

   sftp -o ConnectTimeout=10 rcm_user@sftp.mits.example.gov

   Result	Likely cause	Next step
   Connection refused / ETIMEDOUT	Network or partner down	Wait, then escalate to partner / network team
   Permission denied (publickey,password)	Credential expired or rotated upstream	Rotate via credential vault
   Lands in shell	App-side issue	Continue to step 4

3. Inspect connection config for typos or stale entries:

   SELECT name, connection_config
   FROM rcm_master.trading_partner
   WHERE partner_code = 'OH_MITS';

   Common drift: port changed, remotePath renamed, hostname migrated. Update via the trading partner UI — direct SQL edits skip the audit trail.

4. If credential rotation is needed, follow Rotate trading-partner credentials. The UI confirms the bind with a Run test button after rotation.

5. Retry failed transactions once connectivity is restored:

   # Replay one transaction
   curl -X POST http://localhost:3011/edi/admin/replay/<edi_transaction_id>
   
   # Or replay a batch
   curl -X POST http://localhost:3011/edi/admin/replay/batch \
     -H "Content-Type: application/json" \
     -d '{"trading_partner_id": "<id>", "since": "2026-04-27T00:00:00Z"}'

6. Monitor retry counts — anything > 3 retries on the same transaction needs manual review (the auto-retry logic gives up at 5 by default):

   SELECT edi_transaction_id, retry_count, error_details
   FROM billing.edi_transaction
   WHERE retry_count > 3 AND status = 'ERROR';

Validation

Check	Expected
sftp from rcm-core host	Lands in shell
EDI transactions in SUBMITTING	Drop back to baseline
Response poller pulling files	New files within poll interval
error_details on retried transactions	Cleared on success

Troubleshooting

Symptom	Cause	Fix
Probe works from your laptop, fails from rcm-core host	Outbound firewall / NSG rules differ	Update Azure NSG; whitelist partner IPs.
auth failed despite a fresh password	Partner expects key auth, not password	Switch to private-key auth in connection_config.credentials.
All partners failing simultaneously	Outbound proxy or DNS broken	Check rcm-core logs for DNS errors; restart if proxy is wedged.
Replay returns 200 but transaction stays ERROR	Underlying batch is malformed (not a transport issue)	Inspect error_details; this is no longer an SFTP problem.
Partner says they received our 837 but our submission_status shows ERROR	Partner ack didn't come back; we don't know we succeeded	Check 999/TA1 response; the partner's portal may show the receipt.

Cross-references

• Trading partner credential vault — for any rotation work.
• Submission routing rules — to check which partners + companion guides are in play.
• Distributed tracing — for correlating SFTP errors across services via trace IDs.

Next

2.4 — 278 prior auth playbook