Reconciliation
Two systems answer two different questions, and they are not the same question.
The local check compares Aurora Meter Pro against itself. It is
AuroraMeter.Pro.Reconcile.run/1, it costs one query, it needs no network, and
it finds exactly one class of problem: an acknowledgement ledger claiming to
have reported more than the counter it reported ever held, which is physically
impossible and means something is wrong locally.
The provider check compares Aurora Meter Pro against Stripe. It is
AuroraMeter.Pro.Outbox.Reconciler, an Oban worker you add to your crontab, and
it is the only thing in this package that asks the provider what it thinks
happened.
config :my_app, Oban,
plugins: [
{Oban.Plugins.Cron,
crontab: [{"*/10 * * * *", AuroraMeter.Pro.Outbox.Reconciler}]}
]
Without that line you get no confirmations and no findings, which is what 0.3.0 did, so nothing regresses. What you do not get is any evidence that Stripe agrees with you.
What it writes, and what it will never write
Two things, and nothing else:
-
rows in
aurora_meter_reconciliation_items, which are statements about a difference and the action a human has to consider; -
the
accepted -> confirmedtransition on outbox items whose quantity a provider aggregate accounts for.
It never edits a counter, an event, a usage report, an outbox item’s financial
content, or a credit row. Refund and dispute reconciliation is a different
system in AuroraMeter.Pro.Credits and this one never calls into it.
Nothing here resolves a finding on its own. resolve_item/3 is a statement that
a human looked, it requires them to say what they did, and it writes an
aurora_meter_recovery_actions row naming them.
What confirmed means, exactly
A Stripe meter event summary is an aggregate. It says what a customer’s total was on a meter over a window. It does not say which of your events are inside it.
So the strongest honest claim is per group, not per item:
a provider aggregate at least as large as our local total covers this period.
That is what confirmed means. It is not “Stripe has this exact event”, because
no summary can say that, and claiming otherwise would make confirmed a lie that
an audit would eventually expose.
What it explicitly does not mean is “no error report arrived”. Absence of an error is not evidence. No item is ever confirmed without a number.
The four provider states
They are reported separately and none is inferred from another.
| State | Observed by | Where you see it |
|---|---|---|
request_accepted |
the outbox item reaching accepted |
the item’s state |
meter_event_counted |
a meter event summary accounting for the group’s total |
the item’s state becoming confirmed; detail.provider_state on a summary finding |
invoice_finalized |
an invoice for the customer whose status has left draft |
detail.provider_state on an invoice_status finding |
invoice_paid |
that invoice’s status being paid |
the same |
Invoice checking is off by default (:reconciler_check_invoices), because it
costs one request per customer. An installation that has not turned it on
observes the first two states and says so, rather than guessing the other two.
The six kinds of finding
These are a 1.0 promise. Script against them.
| Kind | Opened when |
expected |
observed |
detail carries |
|---|---|---|---|---|
manual_adjustment |
a correction could not be applied at the provider | the net quantity that should stand | the quantity standing now | customer, meter, account, mode, period, the original identifier, the correction’s event id, the outbox item id, the invoice where known, and the adjustment in words |
summary_shortfall |
a provider summary reports less than the local total | the local total | the provider’s aggregate | customer, meter id, the summary window, the item ids counted |
summary_excess |
a provider summary reports more than the local total | the local total | the provider’s aggregate | the same |
meter_error_report |
Stripe reported invalid meter events | the item’s quantity, where one matched | 0 | the meter, Stripe’s error count and codes, the sampled identifiers, and which of them matched nothing |
unconfirmed_item |
an item has been accepted past :confirmation_grace_seconds with nothing accounting for it |
the item’s quantity | none | the item id, when it was accepted, the customer and meter |
invoice_status |
an invoice for the customer has left draft |
none | none | the invoice id, its status, and which of the two provider states it is |
One finding is one row. A reconciler polling every ten minutes for a week
opens one row, not a thousand: the identity of a finding is
(kind, tenant_key, feature, period_start, subject_ref) and the database
enforces it.
A summary_excess also confirms the group. The two are not in conflict: your
items are covered by an aggregate larger than them, and there is more on that
meter than you put there, which is usually another writer and is a real
operational finding in its own right.
What reconciliation cannot prove
Read this before you put a green dashboard in front of anyone.
- A summary proves an aggregate, not an item. See above.
- Absence of an error report is not evidence. Stripe sends one when it finds invalid events; it sends nothing at all when it is not looking.
-
Stripe samples. A
v1.billing.meter.error_report_triggeredbody carries an error count and a handful ofsample_errors, never the full list of rejected events. The identifiers in ameter_error_reportfinding are a sample; theerror_countis the number that matters. -
The error report only arrives if you subscribed to it. It is a thin event
on Stripe’s v2 event stream, so it needs an event destination with the Thin
payload style. A classic webhook endpoint receives nothing, and an installation
that never created one will see no
meter_error_reportfindings for ever. That is not evidence of health. - Invoice status is only checked if you turned it on.
-
A feature needs a meter id to be reconciled at all. Sending a meter event
takes the meter’s event name (
:stripe_meters); reading a summary takes the meter’s id in the path (:stripe_meter_ids). A feature with the first and not the second is exported and never reconciled. The reconciler logs it and counts it; it does not pass over it in silence. - A provider that did not answer is not a difference. A failed summary read confirms nothing and opens nothing.
-
The reconciler is bounded. One run makes at most
:reconciler_max_requestsrequests, examines at most:reconciler_max_groupsgroups and runs for at most:reconciler_time_budget_ms. A large installation will not reconcile everything in one run, and the lag grows with tenant count.
Corrections, and the 24 hour window
AuroraMeter.correct/4 records the credit as a fact and stages it for export.
Getting it to Stripe is two requests, because Stripe’s meter event adjustment API
cancels a whole event by its identifier and has no partial adjustment:
- cancel the identifier currently standing for that fact;
- if the net is greater than zero, send the net under a derived identifier.
Both idempotency keys come from the item’s own immutable identifier, so a retry repeats both and Stripe deduplicates each. A crash between them under-reports until the retry lands, which is the safe direction. No negative meter event is ever fabricated: Stripe reports a negative cycle total as a quantity of zero anyway, and core’s cumulative bound already guarantees the net is non-negative.
Stripe documents, on cancel.identifier, that
You can only cancel events within 24 hours of Stripe receiving them.
That window runs from receipt, not from when the usage happened. So a
correction of an event Aurora Meter sent more than
:adjustment_window_seconds ago (23 hours by default, an hour under the
documented 24) cannot be applied at all.
Expect that to be the common case for anything but a prompt correction. When
it happens the item becomes quarantined with
quarantine_reason: "provider_window_closed" and one manual_adjustment finding
is opened carrying every number needed to make the adjustment by hand. Both
writes happen in one transaction: a quarantine without the numbers is a credit
nobody can make.
Three more quarantine reasons mean the same thing to you:
| Reason | Means |
|---|---|
original_not_exported |
the corrected fact was never staged for export, so there is nothing at the provider to cancel |
correction_chain_exhausted |
the fact was already corrected to zero at the provider |
correction_chain_unresolved |
the previous correction of that fact was itself quarantined, so what is standing at the provider is not known |
Reading and closing findings
# What is open, newest first.
AuroraMeter.Pro.Reconcile.list_items(state: "open", limit: 50)
# One tenant, one kind.
AuroraMeter.Pro.Reconcile.list_items(tenant: org, kind: "manual_adjustment")
# Close one, saying what you did and who you are.
AuroraMeter.Pro.Reconcile.resolve_item(
item.id,
"Credited 3 units on invoice in_TEST by hand; ticket SUP-42",
actor: "liam"
)
Resolving changes nothing else. No counter moves, no outbox item moves, no credit
is granted. It records that a human looked, and the aurora_meter_recovery_actions
row is the audit trail for that.
Telemetry
| Event | Measurements | Metadata |
|---|---|---|
[:aurora_meter, :pro, :reconciliation, :run] |
duration_ms, requests, groups, confirmed, items_opened |
stopped (:complete, :requests, :groups or :time) |
[:aurora_meter, :pro, :reconciliation, :difference] |
expected, observed |
kind, tenant_key, feature |
[:aurora_meter, :pro, :correction] |
count |
outcome, subject_ref, item_id, window, net_value, original_age_seconds |
stopped is worth a gauge. A run that keeps ending at :requests or :time is
an installation whose confirmation lag is growing.