Reconciliation

Two systems answer two different questions, and they are not the same question.

The local check compares Aurora Meter Pro against itself. It is AuroraMeter.Pro.Reconcile.run/1, it costs one query, it needs no network, and it finds exactly one class of problem: an acknowledgement ledger claiming to have reported more than the counter it reported ever held, which is physically impossible and means something is wrong locally.

The provider check compares Aurora Meter Pro against Stripe. It is AuroraMeter.Pro.Outbox.Reconciler, an Oban worker you add to your crontab, and it is the only thing in this package that asks the provider what it thinks happened.

config :my_app, Oban,
  plugins: [
    {Oban.Plugins.Cron,
     crontab: [{"*/10 * * * *", AuroraMeter.Pro.Outbox.Reconciler}]}
  ]

Without that line you get no confirmations and no findings, which is what 0.3.0 did, so nothing regresses. What you do not get is any evidence that Stripe agrees with you.

What it writes, and what it will never write

Two things, and nothing else:

  • rows in aurora_meter_reconciliation_items, which are statements about a difference and the action a human has to consider;
  • the accepted -> confirmed transition on outbox items whose quantity a provider aggregate accounts for.

It never edits a counter, an event, a usage report, an outbox item’s financial content, or a credit row. Refund and dispute reconciliation is a different system in AuroraMeter.Pro.Credits and this one never calls into it.

Nothing here resolves a finding on its own. resolve_item/3 is a statement that a human looked, it requires them to say what they did, and it writes an aurora_meter_recovery_actions row naming them.

What confirmed means, exactly

A Stripe meter event summary is an aggregate. It says what a customer’s total was on a meter over a window. It does not say which of your events are inside it.

So the strongest honest claim is per group, not per item:

a provider aggregate at least as large as our local total covers this period.

That is what confirmed means. It is not “Stripe has this exact event”, because no summary can say that, and claiming otherwise would make confirmed a lie that an audit would eventually expose.

What it explicitly does not mean is “no error report arrived”. Absence of an error is not evidence. No item is ever confirmed without a number.

The four provider states

They are reported separately and none is inferred from another.

State Observed by Where you see it
request_accepted the outbox item reaching accepted the item’s state
meter_event_counted a meter event summary accounting for the group’s total the item’s state becoming confirmed; detail.provider_state on a summary finding
invoice_finalized an invoice for the customer whose status has left draft detail.provider_state on an invoice_status finding
invoice_paid that invoice’s status being paid the same

Invoice checking is off by default (:reconciler_check_invoices), because it costs one request per customer. An installation that has not turned it on observes the first two states and says so, rather than guessing the other two.

The six kinds of finding

These are a 1.0 promise. Script against them.

Kind Opened when expected observed detail carries
manual_adjustment a correction could not be applied at the provider the net quantity that should stand the quantity standing now customer, meter, account, mode, period, the original identifier, the correction’s event id, the outbox item id, the invoice where known, and the adjustment in words
summary_shortfall a provider summary reports less than the local total the local total the provider’s aggregate customer, meter id, the summary window, the item ids counted
summary_excess a provider summary reports more than the local total the local total the provider’s aggregate the same
meter_error_report Stripe reported invalid meter events the item’s quantity, where one matched 0 the meter, Stripe’s error count and codes, the sampled identifiers, and which of them matched nothing
unconfirmed_item an item has been accepted past :confirmation_grace_seconds with nothing accounting for it the item’s quantity none the item id, when it was accepted, the customer and meter
invoice_status an invoice for the customer has left draft none none the invoice id, its status, and which of the two provider states it is

One finding is one row. A reconciler polling every ten minutes for a week opens one row, not a thousand: the identity of a finding is (kind, tenant_key, feature, period_start, subject_ref) and the database enforces it.

A summary_excess also confirms the group. The two are not in conflict: your items are covered by an aggregate larger than them, and there is more on that meter than you put there, which is usually another writer and is a real operational finding in its own right.

What reconciliation cannot prove

Read this before you put a green dashboard in front of anyone.

  • A summary proves an aggregate, not an item. See above.
  • Absence of an error report is not evidence. Stripe sends one when it finds invalid events; it sends nothing at all when it is not looking.
  • Stripe samples. A v1.billing.meter.error_report_triggered body carries an error count and a handful of sample_errors, never the full list of rejected events. The identifiers in a meter_error_report finding are a sample; the error_count is the number that matters.
  • The error report only arrives if you subscribed to it. It is a thin event on Stripe’s v2 event stream, so it needs an event destination with the Thin payload style. A classic webhook endpoint receives nothing, and an installation that never created one will see no meter_error_report findings for ever. That is not evidence of health.
  • Invoice status is only checked if you turned it on.
  • A feature needs a meter id to be reconciled at all. Sending a meter event takes the meter’s event name (:stripe_meters); reading a summary takes the meter’s id in the path (:stripe_meter_ids). A feature with the first and not the second is exported and never reconciled. The reconciler logs it and counts it; it does not pass over it in silence.
  • A provider that did not answer is not a difference. A failed summary read confirms nothing and opens nothing.
  • The reconciler is bounded. One run makes at most :reconciler_max_requests requests, examines at most :reconciler_max_groups groups and runs for at most :reconciler_time_budget_ms. A large installation will not reconcile everything in one run, and the lag grows with tenant count.

Corrections, and the 24 hour window

AuroraMeter.correct/4 records the credit as a fact and stages it for export. Getting it to Stripe is two requests, because Stripe’s meter event adjustment API cancels a whole event by its identifier and has no partial adjustment:

  1. cancel the identifier currently standing for that fact;
  2. if the net is greater than zero, send the net under a derived identifier.

Both idempotency keys come from the item’s own immutable identifier, so a retry repeats both and Stripe deduplicates each. A crash between them under-reports until the retry lands, which is the safe direction. No negative meter event is ever fabricated: Stripe reports a negative cycle total as a quantity of zero anyway, and core’s cumulative bound already guarantees the net is non-negative.

Stripe documents, on cancel.identifier, that

You can only cancel events within 24 hours of Stripe receiving them.

That window runs from receipt, not from when the usage happened. So a correction of an event Aurora Meter sent more than :adjustment_window_seconds ago (23 hours by default, an hour under the documented 24) cannot be applied at all.

Expect that to be the common case for anything but a prompt correction. When it happens the item becomes quarantined with quarantine_reason: "provider_window_closed" and one manual_adjustment finding is opened carrying every number needed to make the adjustment by hand. Both writes happen in one transaction: a quarantine without the numbers is a credit nobody can make.

Three more quarantine reasons mean the same thing to you:

Reason Means
original_not_exported the corrected fact was never staged for export, so there is nothing at the provider to cancel
correction_chain_exhausted the fact was already corrected to zero at the provider
correction_chain_unresolved the previous correction of that fact was itself quarantined, so what is standing at the provider is not known

Reading and closing findings

# What is open, newest first.
AuroraMeter.Pro.Reconcile.list_items(state: "open", limit: 50)

# One tenant, one kind.
AuroraMeter.Pro.Reconcile.list_items(tenant: org, kind: "manual_adjustment")

# Close one, saying what you did and who you are.
AuroraMeter.Pro.Reconcile.resolve_item(
  item.id,
  "Credited 3 units on invoice in_TEST by hand; ticket SUP-42",
  actor: "liam"
)

Resolving changes nothing else. No counter moves, no outbox item moves, no credit is granted. It records that a human looked, and the aurora_meter_recovery_actions row is the audit trail for that.

Telemetry

Event Measurements Metadata
[:aurora_meter, :pro, :reconciliation, :run] duration_ms, requests, groups, confirmed, items_opened stopped (:complete, :requests, :groups or :time)
[:aurora_meter, :pro, :reconciliation, :difference] expected, observed kind, tenant_key, feature
[:aurora_meter, :pro, :correction] count outcome, subject_ref, item_id, window, net_value, original_age_seconds

stopped is worth a gauge. A run that keeps ending at :requests or :time is an installation whose confirmation lag is growing.