Alerts
These are worked deployment examples, not service level objectives, not defaults, and not thresholds this project can set for your load. Every number below is derived from a configuration value you control, and the derivation is shown so you can recompute it for your own settings. Nothing here is enforced by the package.
Telemetry is the event catalogue every signal comes from.
Aurora Meter’s own docs/alerts.md carries the free half (buffered backlog,
flush errors, aged holds, cluster divergence, event rejections); this page is
the commercial half, where the thing at stake is money that has left one system
and not arrived in the other.
The values the derivations assume, all from Operations and your own Oban crontab:
| Value | Typical | What it means here |
|---|---|---|
| the deliverer’s cron interval | 1 minute | how often the export outbox is drained |
uncertainty_horizon_seconds |
82,800 (23 hours) | how long a provider operation may stay unresolved before a human must look |
| the reconciler’s cron interval | daily | how often Pro compares what it sent against what the provider shows |
1. Export backlog age
Signal. aurora_meter.pro.outbox.gauge.oldest_pending_age_seconds above
2 x the deliverer’s cron interval.
Derivation. A healthy deliverer empties the due items every run, so twice the interval is the smallest bound a single slow run cannot cross. With a one-minute cron: above 120 seconds.
Severity. Warning.
Runbook. Outbox.
Also alert on silence. The gauge is emitted only when the aggregate query
succeeds, and a paused deliverer emits nothing rather than a zero. That is
deliberate: a zero backlog from a queue that is not running is the most
misleading number this package could produce. So alert on the absence of
[:aurora_meter, :pro, :outbox, :gauge] for more than a few intervals as well
as on its value.
2. Any uncertain operation
Signal. aurora_meter.pro.outbox.gauge.uncertain above zero.
Derivation. An uncertain item is by definition one no automated retry can resolve: the provider was changed and Pro cannot tell how far. It exists to be looked at, so the threshold is zero and there is nothing to tune.
Severity. Page during business hours.
Runbook. Recovery.
3. Any reconciliation difference
Signal. Any [:aurora_meter, :pro, :reconciliation, :difference] with a
non-zero observed - expected.
Derivation. The reconciler compares what Pro sent against what the provider shows. A difference is either a bug in the export path or a change somebody made in the provider’s own dashboard; neither heals itself, and both end up on an invoice.
Severity. Page.
Runbook. Reconciliation.
4. Quarantine growth
Signal. aurora_meter.pro.outbox.gauge.quarantined increasing between
samples.
Derivation. Quarantine means a mapping, mode or window problem: an item that cannot be sent as it stands, whatever is retried. It never clears itself, so the signal is the rate of change rather than the level. A non-zero level that is not moving is a backlog somebody has already seen.
Severity. Warning.
Runbook. Operations.
5. A pending payment past the horizon
Signal. A credit account holding pending_payment_intent_id for longer than
uncertainty_horizon_seconds.
Derivation. The horizon is how long Pro will wait for a provider operation to resolve itself before a human has to decide. Past it, money may have left the customer’s card without credit being granted, which is the one failure in this package a customer notices immediately.
The operator page reports this figure directly, with the exact
AuroraMeter.Pro.Recovery.attach_payment_intent/2 call to paste beside it.
Severity. Page.
Runbook. Recovery.
6. Provider latency, and what it is not
Signal. The distribution of [:aurora_meter, :pro, :provider, :stop]‘s
duration, split by result.
Derivation. There is none, and that is the point: this is a chart rather than an alert. One span is one HTTP attempt, so the distribution is the provider’s own latency and not Pro’s retry policy. Alert on the backlog above, which is the thing you can act on; a slow provider that is nonetheless keeping up is not an incident.
Do not alert on this
- A retryable delivery failure. The item is retried under the same identifier and the provider deduplicates it. Alert on the backlog age, which is what a retry that is not converging looks like.
-
A
:duplicatewebhook. Receiving the same provider event twice is the normal behaviour of every webhook system, and the dedupe record is the feature working. - A non-zero quarantine level that is not moving. See rule 4.
-
[:aurora_meter, :pro, :provider, :exception]on its own. One exception is a network. Three in a row is rule 1 or rule 2, depending on which state the items landed in.