Recovering uncertain provider operations

A timeout is not evidence of rejection. Pro preserves the original operation and stops automatic replay after 23 hours when Stripe may no longer retain its idempotency key. A known PaymentIntent can still be retrieved safely. These procedures are for an administrator with access to the host database and the matching Stripe account and mode. Establish what happened at the provider first, then record it here.

Everything below goes through AuroraMeter.Pro.Recovery or mix aurora_meter_pro.recovery. The console recipes that used to be on this page are gone: they wrote with repo.update! after asserting in memory, which bypasses every conditional predicate the outbox rests on, and one of them took a lock the reporter has not held since the outbox landed, so it appeared to serialise while serialising nothing.

This page is the procedure, item by item. Section 7 of Operations is the shorter form: the look, inspect, act shape every one of these shares, and the four properties that hold for all of them. Section 8 of the same page is why an advisory lock is not how you stop a worker.

What the software will and will not do for you

It will not decide anything. An export whose outcome is unknown stays uncertain, a payment whose PaymentIntent is unknown stays pending, and an item whose mapping is wrong stays quarantined, for ever, until a human says what they found.

It also cannot verify what you tell it. Acknowledge an item that never reached Stripe and that usage is treated as billed and never sent again: the customer is under-billed and nothing here will notice. Abandon an item that did reach Stripe and the window is staged again under a new identity: the customer is billed twice.

The asymmetry points one way. Over-billing is the worse error, so when you are not sure, leave the item uncertain. Uncertain work is visible, queryable and alarmable. A wrong abandonment is a refund and an apology.

What the software does instead of trusting you blindly:

• it refuses an acknowledgement that carries no provider reference;
• it compares the state you say you are acting on against the row, and refuses when they differ, naming the current values;
• it records who claimed what, why, and on what evidence, applied or refused;
• it leaves the reconciler running, which will contradict a wrong claim with a number out of Stripe’s own meter event summaries.

There is no force: option and there never will be. An operator who cannot fill in the expected-state map does not yet understand the state well enough to change it.

Authorisation is yours. AuroraMeter.Pro.Recovery and the Mix task assume an already authorised operator; the package cannot authenticate anyone. Putting this module behind a web route without your own authorisation in front of it is your mistake, not a missing feature. This package ships no recovery web surface at all. evidence is free text a human wrote, stored as given: treat it as untrusted if you ever render it.

Find pending work

mix aurora_meter_pro.recovery list
mix aurora_meter_pro.recovery list --state quarantined --kind event --older-than 24h
mix aurora_meter_pro.recovery payments

Or in an iex session:

AuroraMeter.Pro.Recovery.list_uncertain(state: :all, limit: 20)
AuroraMeter.Pro.Recovery.pending_payments()

list_uncertain/1 answers uncertain, rejected and quarantined items by default, filtered by :tenant, :kind, :older_than and :limit. pending_payments/1 is every credit account with a charge in flight.

Pending work can exist on disabled or above-threshold accounts. Do not clear it to bypass an eligibility guard. Capture the payload, identifier and time, and check failed and discarded Oban jobs alongside Stripe request logs and events.

Look at one item before you touch it

mix aurora_meter_pro.recovery inspect ITEM_ID
mix aurora_meter_pro.recovery inspect ITEM_ID --format json

{:ok, detail} = AuroraMeter.Pro.Recovery.inspect_item(item_id)

It returns the row, the frozen payload, the exporter’s own limits (describe/0), whether the first attempt is still inside the idempotency horizon, the aurora_meter_usage_reports row the item is staged against, any open reconciliation items naming the same subject, the eligibility the item would be re-checked against today, and the recovery actions already recorded against it.

Read that last section. An item three people have already tried to acknowledge is not an item that needs a fourth attempt; it is an item whose state somebody is misreading.

--format json exists so a run can be pasted into an incident record verbatim.

Resolve a payment whose intent is known

If pending_payment_intent_id is populated, call AuroraMeter.Pro.Credits.AutoTopUpWorker.attempt(tenant_key). It retrieves the existing intent; it does not create another charge. A succeeded intent grants its received amount once, a definitive failure resolves the attempt, and a processing or action-required intent stays pending. Do not repeatedly create new attempts to work around customer authentication requirements.

Attach a PaymentIntent you have identified

If the intent id was lost, find the original request in Stripe using pending_idempotency_key or the intent’s metadata.auto_top_up_key. Retrieve the current intent and verify all of the following against the stored payload:

• Stripe account and mode, tenant metadata and auto_top_up_key;
• customer, payment method, currency and original requested amount;
• the intent belongs to this specific operation, not a separate manual top-up.

Only then attach it. The key and the payload stay intact.

mix aurora_meter_pro.recovery attach-intent TENANT \
  --expected-key aurora_meter:auto_top_up:... --intent pi_... \
  --reason TEXT --evidence TEXT --yes

AuroraMeter.Pro.Recovery.attach_payment_intent("org:example",
  expected: %{pending_idempotency_key: "aurora_meter:auto_top_up:example"},
  intent_id: "pi_example",
  reason: "identified the intent in the Stripe dashboard",
  evidence: "Dashboard search on the idempotency key returned pi_example at 09:12Z"
)

It refuses an id that does not start with pi_, an expected key that does not match the account, and an account that already carries a different intent. It writes one column, under the same auto_top_up:<tenant_key> advisory lock the worker and the webhook take, and then resumes the payment through the GET-only path above. The resume’s answer is written into the audit row’s outcome. Pass resume: false (or --no-resume) to attach without resuming.

The grant itself is still keyed on the intent id by the ledger, so attaching an intent that was already granted is a no-op. Attaching the wrong intent grants the wrong amount, which is why the verification list above is not optional.

Abandon a payment that was never created

If no intent can be identified, leave it pending until Stripe records or support establish the outcome. An empty search or a zero invoice is not sufficient proof of no charge. Only a verified non-created or rejected operation may be abandoned.

mix aurora_meter_pro.recovery abandon-payment TENANT \
  --expected-key aurora_meter:auto_top_up:... --reason TEXT --evidence TEXT --yes

AuroraMeter.Pro.Recovery.abandon_payment("org:example",
  expected: %{
    pending_idempotency_key: "aurora_meter:auto_top_up:example",
    pending_payment_intent_id: nil
  },
  reason: "Stripe support confirmed no PaymentIntent was created",
  evidence: "Support case 4412: no object exists for this idempotency key"
)

It refuses with {:error, :intent_known} when an intent is known, from either side: yours or the account’s. Abandoning a known intent throws away the one thing that can resolve it.

It clears the four pending columns and nothing else. It does not reset last_attempt_at, does not change failure_count or disabled_reason, and grants, reverses and restores nothing: the core credit ledger is not reachable from the recovery module at all. Do not reset its timestamp or reuse the expired key as a way to authorise a new charge.

A payment refused because its customer could not be verified

A paying event may only credit the tenant its metadata names when the Stripe customer it names is one this installation can stand behind. Two things are refused, both answered 200 so Stripe stops retrying, both logged at error naming the tenant and the customer, and neither grants anything or writes anything to the account:

• unknown_customer: the customer is not in this installation’s Stripe account. This is what a webhook endpoint pointed at the wrong installation looks like, and what stale or hand-edited payment data looks like.
• customer_belongs_to_another_tenant: the customer is already on record against a different tenant. One tenant’s payment arriving with another tenant’s name on it.

A tenant moving between its own Stripe customers is not refused, and neither is an event that names no customer at all. A provider that did not answer is not refused either: that is {:error, :customer_unverified} and a 500, so Stripe redelivers it.

The refused event is deliberately left unapplied in aurora_meter_webhook_events, so it is visible as unfinished work and so the recovery below can still run it.

What to do

1. Read the log line. It names the event type and id, the tenant and the customer, and which of the two refusals it was.
2. Establish at Stripe what the payment actually is: the account and mode, the PaymentIntent, its amount and currency, its metadata.tenant_key, and which customer paid. A refusal is usually correct. Do not continue until you can say which tenant’s money this is.
3. If the payment is genuine and the record was wrong, correct the record, then resend the event from Stripe (stripe events resend evt_..., the same request Stripe’s own retry makes). It was never marked applied, so it runs again and grants normally. This is the only route that keeps the ledger keyed on the PaymentIntent, and it is therefore the only one that cannot double grant.
4. If the event can no longer be resent, grant it directly on its PaymentIntent id, which is idempotent per reference:

AuroraMeter.Pro.Credits.record_payment("org:example", "pi_example", 2_500,
  %{source: "top_up", recovered: "refused customer, verified in the dashboard"})

Use the PaymentIntent id as the reference and nothing else. Any other string is a reference the ledger has never seen, so a later redelivery of the real event grants the same money a second time.

5. If the refusal was right, there is nothing to do here: the money belongs to another tenant, or to no tenant this installation knows. Fix whatever produced the event.

Nothing in AuroraMeter.Pro.Recovery grants credit, and this procedure does not change that: step 3 hands the grant back to the ordinary webhook path, and step 4 is the host’s own ledger API with the provider’s own reference.

Resolve an old usage export

Use the payload’s identifier to locate the exact request. Check customer, event name, timestamp and value. Stripe aggregates asynchronously and can issue meter error events after accepting a request. Verify the final outcome against the meter records and error events; a request receipt alone does not prove billing.

Verified accepted

mix aurora_meter_pro.recovery acknowledge ITEM_ID \
  --lease-token N --subject-ref REF --provider-ref evt_... \
  --reason TEXT --evidence TEXT --yes

AuroraMeter.Pro.Recovery.acknowledge_item(item_id,
  expected: %{
    state: :uncertain,
    lease_token: 3,
    subject_ref: "usage:api_calls:2026-09-01T00:00:00Z:0:5",
    provider_ref: "evt_example"
  },
  reason: "verified at the provider",
  evidence: "Stripe request log req_example at 2026-09-15T09:12Z: 200 for this identifier"
)

The item moves to accepted and your provider reference is written to it. For a buffered window it also advances usage_reports.last_reported to the staged value and clears the staged link, in the same transaction, because a recovery acknowledgement is an acknowledgement: leaving the ledger behind means the next reporter run recomputes the same delta and stages it again under a new identity, which is a second charge.

expected.provider_ref is required and must carry text. An acknowledgement with no provider reference is an assertion with nothing behind it.

The item does not move to confirmed. accepted means the request landed, which is what you are claiming. confirmed means the reconciler has seen Stripe’s own aggregate account for it, and only the reconciler may say that.

Verified rejected

mix aurora_meter_pro.recovery abandon ITEM_ID \
  --lease-token N --subject-ref REF --reason TEXT --evidence TEXT --yes

The item becomes rejected, which is terminal and visible. For a buffered window, last_reported is left exactly where it is and the staged link is cleared, so the next reporter run stages the remaining usage with a fresh identifier.

That restage is a send. Before you abandon, check Stripe’s timestamp acceptance window and whether the invoice is still open to usage changes. An already finalized invoice may need a separate billing adjustment; never represent such an adjustment as a meter report that did not occur.

The mapping was wrong, and is fixed

mix aurora_meter_pro.recovery requeue ITEM_ID \
  --lease-token N --subject-ref REF --quarantine-reason feature_unmapped \
  --reason TEXT --evidence TEXT --yes

A quarantined item goes back to pending and is sent on the next deliverer run. Two refusals happen before anything is written:

• {:error, :mapping_changed} when the item’s mapping_version is not the configured one. An item staged under a mapping that no longer exists must not be resurrected into a different one; the producer stages a new item under the new mapping.
• {:error, {:still_ineligible, check}} when re-running the eligibility chain on the item still fails, naming the check.

Three of those checks have no equivalent when an item is first staged, and they are the common answer here: :exporter_changed, :destination_account_changed and :mode_changed. Those three columns are part of an item’s identity and are immutable in the row and in the database, so an item quarantined as :no_account is still addressed to the literal "unconfigured" after you configure the account. Fixing the configuration means a new item, not this one.

Moving a cutover watermark

AuroraMeter.Pro.Cutover.schedule/3 refuses to move a watermark that already exists, because moving one changes what is billable. The guarded path is here:

AuroraMeter.Pro.Recovery.reschedule_cutover(:api_calls,
  expected: %{effective_period_start: ~U[2026-10-01 00:00:00Z]},
  to: ~U[2026-11-01 00:00:00Z],
  reason: "the migration slipped a month",
  evidence: "Change record CR-4412 moved the deploy to the following cycle"
)

It refuses with {:error, {:already_effective, counts}} once anything has been billed at or after the earlier of the two watermarks: an event item staged for such a period, or a usage report with last_reported > 0. Moving a watermark past work that has been reported reassigns that period from one reporting path to the other, which bills it twice or not at all. Correct it before anything runs, or leave it and correct the affected periods at the provider by hand.

Every operation requires a reason and evidence

reason: says why you are changing the record. evidence: says what you saw that makes the change correct: a Stripe request id, an event id, a dashboard search and its result, a support case.

Neither has a default. evidence must be at least 16 characters and must not repeat reason. That floor is crude on purpose: it makes “ok” and “done” impossible without pretending to judge the quality of a human’s sentence, which no library can do.

A mutating Mix subcommand without both exits non-zero saying why. Without --yes it prints the row, the change it would make and the expected-state map it would send, and writes nothing.

What the audit trail records, and for how long

Every mutating call writes exactly one aurora_meter_recovery_actions row, whether it applied or was refused. Three operators being refused on one item is exactly the signal that something is being misread, and a table that recorded only successes would say one operator acted.

Read it with mix aurora_meter_pro.recovery actions --subject SUBJECT or AuroraMeter.Pro.Recovery.actions/2.

Rows are kept for a year by default and pruned by the retention worker. Nothing in the table is ever updated except outcome on an attach, which gains its resume’s answer when the resume returns.

Reads write nothing, not even an audit row. Who read what is your application’s logging concern; an audit table that fills up with reads stops being one.

One gap, stated rather than hidden

The two payment operations run inside a session advisory lock on a pinned connection and therefore open no transaction: the existing payment code never does inside that lock, and it must not start. So a crash between the conditional update and the audit row leaves the change applied and unrecorded. The window is two local statements wide and the change is visible on the account (the intent id is on it), but it is a gap and this is where it is written down. Closing it would need a transaction inside the pinned-connection lock, which is a worse trade.

The three outbox operations have no such gap: the update, the ledger statement and the audit row are one transaction.

Verify the recovery

Check that resolved pending state is cleared, payment ledger references are unique, balances reflect actual money, and the resulting invoice matches usage and price tiers. Run the worker and the deliverer again and verify they add no duplicate money or usage. Preserve the original evidence and any manual changes.

AuroraMeter.Pro.Reconcile.run/1 checks only local over-reporting against persisted counters. It cannot verify Stripe receipt, invoice payment or missing unreported usage: AuroraMeter.Pro.Outbox.Reconciler is the half that asks Stripe, and Reconciliation is its page. See also Usage reporting, the outbox and Stripe’s idempotency contract.