When the API Returns 500, Did the Charge Post or Not?

The problem

MSPs reselling telecom services run a monthly cycle: download a usage report, parse it, and push itemized charges into a PSA billing system. At any scale the work is error-prone. Charges land on the wrong agreement. Product mappings drift. A retried sync posts duplicates because nobody knows whether the upstream API accepted the first attempt.

The December 2025 tax-row incident in production made the stakes concrete: a structural identity bug produced hundreds of mismatching billing cycles across a large account set. The reconciliation audit caught it. That is the bar billing software has to meet — not "usually works," but "we can prove what posted."

Preview before sync

The highest-leverage control is not smarter retries. It is refusing to sync blind. Before any write, Meter builds the payload it would send and diffs it against what the billing system already holds for that agreement and report month.

Comparison keys normalize product code, quantity, unit price, and effective date so display-string drift does not surface as false mismatches. Operators see adds, updates, and no-ops in a modal before anything commits. Most months the diff is empty. The months it is not are the ones that save a finance team a week of cleanup.

Ghost additions

ConnectWise (and most enterprise APIs) sometimes returns 5xx on a POST that actually succeeded. Blind retry duplicates the charge. Fail closed and you drop revenue.

The ghost-addition detector queries the live API within a short lookback window and matches by a hash of the invoice description plus product and quantity. If the record exists, the sync marks it succeeded instead of posting again. If it does not, retry proceeds with context. The rule is simple: never guess when money is involved.

DB-level locking

Concurrent sync requests for the same account would race through preview and both commit. Meter acquires a per-account lock as a database row with a primary-key constraint. A duplicate insert means another worker owns the sync — the caller backs off instead of proceeding. Stale locks from crashed workers expire on a timer so deadlocks do not block the next billing window.

Effective-date recovery

A narrower but recurring failure: the billing API rejects additions whose effective date precedes the agreement billing start. That is not a data bug — it is a calendar edge case that used to require operator intervention every month.

The handler recognizes the specific error code, fetches the agreement billing start, adjusts the date, and retries once. Small code path, large reduction in support tickets.

What I would do differently

I would push more of the comparison engine into immutable event logs earlier. The preview diff is powerful but still point-in-time. A ledger of "intent to post" events would make audits even easier to replay. For a solo-owned production system at this scale, the preview + registry pattern was the right trade.