The export outbox

Latest release v1.2.0 · 2026-10-27 · this is the file that ships inside the package

README Changelog GitHub ↗ Aurora Meter page

The export outbox

Everything Aurora Meter Pro sends to Stripe as usage goes through one table, aurora_meter_outbox_items, and one state machine. This page is the machine.

The shape of it is ordinary: a producer writes an immutable row saying what should be sent, and a consumer sends it and records what happened. What makes it worth a page is the set of things it refuses to do, each of which was a way the previous design could bill a customer twice.

The row

Column What it is
tenant_key, subject_kind, subject_ref, exporter, destination_account, mode, mapping_version the identity, unique in the database
payload the financial intent: customer, meter, value, timestamp, identifier, feature, period. Immutable
state pending, leased, accepted, confirmed, uncertain, rejected, quarantined
attempts, next_attempt_at, first_attempt_at how many tries, when the next one is due, when the first one happened
lease_owner, lease_expires_at, lease_token who holds it, until when, and the fence
provider_ref, accepted_at, confirmed_at what the provider called it and when
last_error, quarantine_reason sanitised: a code, a status and at most 200 characters of message

subject_ref for a buffered usage window is "usage:<feature>:<period_start>:<from>:<to>". Both bounds are in it on purpose: restaging an unacknowledged window is the same identity and collapses into the existing row, while a later window after an acknowledgement is a different identity with a different idempotency key.

The idempotency key is a pure function of the identity:

identifier = "aurora_" <> sha256(subject_ref | account | mode | mapping_version)

71 bytes, comfortably inside Stripe’s 100 character limit for a meter event identifier. It carries no nonce, deliberately. The old reporter hashed a fresh UUID into it, because its design re-staged from fresh usage after a rejection and needed a new key for the replacement. In this design rejected is terminal and a new identity comes only from a new subject_ref, so a nonce would not merely be unnecessary: it would make every retry a second charge.

The states

              enqueue (eligible)              enqueue (ineligible)
                     |                                 |
                     v                                 v
 +---------------> pending --lease--> leased --accept--> accepted --confirm--> confirmed
 |                   ^                 | | |
 |  retry(delay)     |                 | | +--reject------> rejected    (terminal, visible)
 +-------------------+                 | +----uncertain---> uncertain <--+ (cap or horizon)
                                       |                       |
                                       +--quarantine--> quarantined --requeue--> pending
                                                                       ^
                                                          uncertain --recovery--+
# From To Trigger Predicate A stale writer sees
T0 none pending producer, eligible INSERT ... ON CONFLICT (identity) DO NOTHING zero rows inserted; it reads the existing id and proceeds with it
T0q none quarantined producer, ineligible the same insert with a reason the same
T1 pending or expired leased leased a deliverer claims SELECT ... FOR UPDATE SKIP LOCKED, then WHERE id = ? AND ((state = 'pending' AND next_attempt_at <= now) OR (state = 'leased' AND lease_expires_at < now)) nothing at all; somebody else has it. Not an error
T2 leased accepted :accepted or {:accepted, ref} WHERE id = ? AND lease_token = ? AND state = 'leased' zero rows, {:error, :stale_lease}, stale_worker telemetry, no retry of the write
T3 leased pending {:retry, delay} inside the cap and the horizon as T2 as T2
T4 leased uncertain :uncertain, or a retry past the cap, or a retry past the horizon as T2 as T2
T5 leased rejected {:rejected, reason} as T2 as T2
T6 leased quarantined a mapping, mode or account problem found at claim time as T2 as T2
T7 accepted confirmed the reconciler has provider evidence WHERE id = ? AND state = 'accepted' zero rows; it re-reads and reports
T8 uncertain accepted, pending or rejected AuroraMeter.Pro.Recovery with an expected-state map WHERE id = ? AND state = 'uncertain' AND lease_token = ? {:error, {:expected_state_mismatch, current}}
T9 quarantined pending an operator requeues after fixing the cause WHERE id = ? AND state = 'quarantined' AND mapping_version = ? {:error, :mapping_changed}

Four properties make the machine safe, and each is there because its absence was a defect.

attempts is incremented at claim time, not at outcome time. A worker that dies between claiming and answering has still consumed an attempt, so a crash loop cannot retry for ever.

first_attempt_at is set on the first claim and never moved. The horizon is measured from it, so it is a property of the item rather than of whichever process happens to be handling it.

rejected is terminal and visible. Nothing in the automatic path leaves it. Only an operator can, and only to record an abandonment.

The attempt cap never produces rejected. Exhausted work becomes uncertain, which is queryable and recoverable. Exhaustion is not evidence that the provider refused anything.

The lease and the fence are two different things

This is the part worth reading twice, because they look alike and only one of them is a correctness property.

Who gets an item is decided by SELECT ... FOR UPDATE SKIP LOCKED inside the claiming transaction. Two workers scanning the same page do not queue behind each other and do not both take a row: the second skips what the first holds. No timestamp is compared to decide it.

Whose write wins is decided by lease_token, a counter incremented on every claim. Every transition out of leased carries AND lease_token = ? AND state = 'leased', so a worker whose item was re-claimed under it writes zero rows and is told so. No timestamp is compared to decide that either.

lease_expires_at decides only when an abandoned item becomes available again. It is the one duration comparison left, and it is a liveness input. The database clock is shared across nodes but it is not monotonic: measured on this hardware, clock_timestamp() stepped backwards nine times in 300 seconds, worst case 439 ms. Against a 120 second lease that is a factor of 273, and the worst a step can do is move a re-claim by 439 ms in either direction. A re-claim at the wrong moment is not a hazard: the previous owner’s write is fenced out, and the resend carries the same identifier, so the provider deduplicates it. There is no arrangement of the clock that produces a double delivery, because no delivery decision reads the clock.

Where a duration genuinely is compared (the lease, the backoff schedule and the idempotency horizon), both sides are clock_timestamp() inside the same statement. Nothing reads the time into Elixir and writes it back.

Retry and backoff

backoff(attempts) = min(cap, base * 2 ^ (attempts - 1)) + jitter

with jitter up to a quarter of the delay. Defaults: base 5 seconds, cap 900 seconds, 20 attempts, which puts the twentieth attempt about four hours after the first and comfortably inside the horizon, so the cap is normally reached first and the horizon is the backstop.

The jitter is not decoration. Without it every item that failed in one batch retries in the same second as every other, which is how a recovering provider is rate limited straight back into failure.

A rate limit uses outbox_rate_limit_delay_seconds (default 30) rather than the exponential backoff, because Stripe’s per-endpoint limit is 25 requests per second and backing off by five seconds is not an answer to being over it.

Quarantine

quarantined is not a hold the system releases on its own. It is an item the software has decided it must not guess about. The reasons:

Reason Means
feature_buffered, attribution_unresolved, original_ineligible core already decided the event is not a candidate
feature_unmapped the feature has no entry in stripe_meters
no_customer the tenant has no provider_customer_id
no_account :stripe_account_id is not configured
mode_mismatch, account_mismatch the item was staged for a different Stripe account or mode from the one this node is configured for
period_buffered the period is before the feature’s cutover watermark, so the buffered reporter owns it
timestamp_window the occurrence is outside what the exporter says the provider accepts

An ineligible event is staged anyway, quarantined, never dropped. That matters most for a correction: a correction that quietly disappears is a customer credited nothing and told otherwise.

Watching it

[:aurora_meter, :pro, :outbox, :gauge] fires once per deliverer run with a count for every state, oldest_pending_age_seconds, and legacy_pending. Alert on uncertain, rejected and quarantined being above zero, and on oldest_pending_age_seconds climbing: all three need a human.

[:aurora_meter, :pro, :outbox, :deliver] fires per item with duration_ms and the state it landed in. It carries no tenant key: that is unbounded cardinality and a metrics backend would either drop the series or bill for it.

[:aurora_meter, :pro, :outbox, :stale_worker] fires when a conditional update affects zero rows. A few are normal on a cluster. A steady stream means leases are expiring while work is still in flight, so raise outbox_lease_seconds.

See usage reporting for how usage gets into the outbox, and recovery for what to do about an item that needs a human.