Telemetry
Aurora Meter Pro emits :telemetry events. It talks to exactly one place,
Stripe, through one client seam, and it sends telemetry nowhere: attaching a
handler, running a reporter and choosing what leaves your network are your
decisions. AuroraMeter.Pro.NoOutboundIoTest asserts that rather than
promising it, with an allow list three files long and a reason beside each.
AuroraMeter.Pro.Telemetry.events/0 is this page in machine-readable form. The
table below is checked against it, and against lib/, on every mix test: an
event that is emitted and not listed here fails the suite, and so does one
listed here and not emitted.
The rules for a metric tag are the core’s
They are stated once, in the free core, and Pro uses them:
AuroraMeter.Telemetry.tag_allow_list/0,
AuroraMeter.Telemetry.forbidden_tags/0 and
AuroraMeter.Telemetry.tag_allowed?/2. One rule written down twice is two rules
waiting to disagree.
In short: a tag is one of result, kind, exporter, state or worker,
plus feature when metrics_feature_label: true. Everything else is
correlation metadata. Pro’s own reasons for that, over and above the core’s:
-
tenant_keyis on every commercial event and is unbounded by definition. -
payment_intent_id,item_id,subject_refandlease_tokenidentify one object each, so a tag on any of them is one time series per object. -
period_startandwatermarkare instants. One series per billing period, for ever. -
attemptsis a count. A count is a measurement, not a dimension.
Where a Pro event’s metadata name is bounded but is not an allow-listed name,
the shipped preset maps it rather than tagging on it raw: outcome becomes
result, subject_kind and action and table become kind, transition
becomes state. The name of a dimension is a contract with a reporter, and the
library keeps that contract closed so a dashboard built on one Aurora Meter
deployment reads the same on another.
AuroraMeter.Telemetry.redact/2 is the supported way to put event metadata in a
log line or a span attribute. It drops every identifier here, including
payment_intent_id and subject_ref, and turns error into error_class.
Three events under a core-looking name
[:aurora_meter, :alert], [:aurora_meter, :usage_report] and
[:aurora_meter, :reconcile] are Pro events in the core’s namespace. A host
attaching a handler cannot tell from the name whether the signal is free or
commercial, and the core’s own telemetry page does not list them, because the
core does not know Pro exists.
They are kept. Renaming them would silently detach every handler already
attached to them, which is a worse outcome than an untidy name, and a rename is
a 2.0 conversation. Every Pro event added since uses [:aurora_meter, :pro, ...].
The events
| Event | Measurements | Metadata | Tags | Since |
|---|---|---|---|---|
[:aurora_meter, :alert] |
percent, threshold, usage |
feature, kind, tenant_key |
kind |
0.1.0 |
[:aurora_meter, :usage_report] |
delta |
feature, period_start, tenant_key |
none | 0.1.0 |
[:aurora_meter, :reconcile] |
drift |
feature, tenant_key |
none | 0.1.0 |
[:aurora_meter, :pro, :auto_top_up] |
amount_cents |
reason, result, tenant_key |
result |
0.3.0 |
[:aurora_meter, :pro, :outbox, :deliver] |
duration_ms |
attempts, outcome, reason, state, subject_kind |
state |
0.4.0 |
[:aurora_meter, :pro, :outbox, :gauge] |
accepted, confirmed, leased, legacy_pending, oldest_pending_age_seconds, pending, quarantined, rejected, uncertain |
none | none | 0.4.0 |
[:aurora_meter, :pro, :outbox, :stale_worker] |
count |
item_id, lease_token, transition |
none | 0.4.0 |
[:aurora_meter, :pro, :credits, :gauge] |
armed, below_threshold, enqueued, reconciliation_required |
operation |
none | 0.4.0 |
[:aurora_meter, :pro, :credits, :wallet_wide_fallback] |
count |
operation, payment_intent_id, tenant_key |
none | 0.6.0 |
[:aurora_meter, :pro, :webhook] |
duration_ms |
api_version, duplicate, outcome, type |
none | 0.4.0 |
[:aurora_meter, :pro, :cutover] |
none |
action, feature, watermark |
none | 0.4.0 |
[:aurora_meter, :pro, :correction] |
count |
item_id, net_value, original_age_seconds, outcome, subject_ref, window |
none | 0.4.0 |
[:aurora_meter, :pro, :reconciliation, :difference] |
expected, observed |
feature, kind, tenant_key |
kind |
0.4.0 |
[:aurora_meter, :pro, :reconciliation, :run] |
confirmed, duration_ms, groups, items_opened, requests |
stopped |
none | 0.4.0 |
[:aurora_meter, :pro, :recovery, :action] |
count |
action, outcome, subject, tenant_key |
none | 0.4.0 |
[:aurora_meter, :pro, :retention, :prune] |
deleted, duration |
blocked, table |
none | 1.0.0 |
[:aurora_meter, :pro, :transition] |
count |
ref, result, tenant_key, to_plan_id, to_version |
result |
1.0.0 |
[:aurora_meter, :pro, :usage_report, :plan_fallback] |
count |
feature, period_start, plan_id, plan_version, tenant_key |
none | 1.0.0 |
[:aurora_meter, :pro, :provider] |
duration, monotonic_time, system_time |
kind, operation, reason, result, stacktrace |
result |
1.0.0 |
Alerting, and the provider span
Alerts is six worked examples over this catalogue, each with the signal, the derivation of its threshold from a value you control, a severity and a runbook link, plus a “do not alert on this” section. They are deployment examples and not service level objectives.
[:aurora_meter, :pro, :provider] is a span around one outbound provider
call. It is emitted at the seam where Pro calls the configured client, so one
span is one HTTP attempt and never the loop that retries it, which is what makes
a latency distribution over these spans mean anything. operation is one of
AuroraMeter.Pro.Telemetry.provider_operations/0, a closed set of callback
names; it is metadata rather than a tag because the tag allow list does not
carry it. No request body, no response body, no customer identifier and no
payment identifier is on the span.
AuroraMeter.OpenTelemetry (core, optional) turns this event and
[:aurora_meter, :pro, :outbox, :deliver] into spans in the host’s own
OpenTelemetry SDK, along with the slow half of core’s catalogue. It attaches to
the Pro event names as data and references no Pro module.
Failure mode, signal, runbook
Core’s docs/telemetry.md carries the same table for the free half: buffered
loss, flush failure, cluster divergence, durable event outcomes, holds, expiry
and recurring grants. This one covers the commercial half.
| State | Signal | Runbook |
|---|---|---|
| Export backlog growing |
pro.outbox.gauge.pending and pro.outbox.gauge.oldest_pending_age_seconds rising |
operations.md section 10 |
| Provider outcome unknown |
pro.outbox.gauge.uncertain above zero; pro.outbox.deliver with an uncertain outcome |
recovery.md |
| Item quarantined by a mapping, mode or window judgement |
pro.outbox.gauge.quarantined above zero |
outbox.md |
| Fenced write from a stale worker |
[:aurora_meter, :pro, :outbox, :stale_worker] |
outbox.md |
| Provider-side difference found |
[:aurora_meter, :pro, :reconciliation, :difference] |
reconciliation.md |
| Reconciliation stopped early |
pro.reconciliation.run metadata stopped |
reconciliation.md |
| Local over-reporting against the provider |
[:aurora_meter, :reconcile] with a non-zero drift |
operations.md section 5 |
| Payment outcome unresolved |
pro.auto_top_up with a failed result; pro.credits.gauge.reconciliation_required above zero |
recovery.md |
| Payment could not be attributed to one lot |
[:aurora_meter, :pro, :credits, :wallet_wide_fallback] |
top-ups.md |
| Guarded recovery action performed |
[:aurora_meter, :pro, :recovery, :action] plus its audit row |
operations.md section 7 |
| Webhook handling failing |
pro.webhook with a failing outcome |
subscriptions.md |
| Paid transition not confirmed by the provider |
[:aurora_meter, :pro, :transition] absent past its effective time |
subscriptions.md |
| A usage row could not be attributed to a recorded plan version |
[:aurora_meter, :pro, :usage_report, :plan_fallback] |
usage-reporting.md |
| Version 10 upgrade incomplete, deliverer cancelling |
pro.outbox.gauge.legacy_pending at 1 |
cutover.md |
| Retention refused to prune |
pro.retention.prune with blocked: true |
retention.md |
| A scheduled worker stopped running | silence from the event that worker emits | scheduler.md |
| Usage alert threshold crossed |
[:aurora_meter, :alert] |
operations.md section 3 |
The backlog gauge, and why absence matters
[:aurora_meter, :pro, :outbox, :gauge] is one event with a measurement per
state, not one event per state. A host writes
last_value("aurora_meter.pro.outbox.gauge.pending") and gets a clean series
per state with no tag at all, which is the cheapest possible answer to the
cardinality question.
AuroraMeter.Pro.Outbox.Deliverer emits one at the end of every run.
AuroraMeter.Pro.Outbox.gauge/0 emits one on demand, for a host whose queue is
paused or that runs the work on another scheduler.
When the aggregate query fails, no sample is emitted at all. Not a zero. A zero on this gauge reads as “the backlog is empty”, and the case it exists for is the one where the backlog is not empty and nobody can see it. The same rule applies to the run itself: a paused deliverer produces no samples, so the last one goes stale and staleness is the signal. Alert on the age of the last sample as well as on its value.
Metrics presets
# mix.exs
{:telemetry_metrics, "~> 1.0"}
defmodule MyApp.Telemetry do
def metrics do
AuroraMeter.Telemetry.Metrics.metrics() ++ AuroraMeter.Pro.Telemetry.Metrics.metrics()
end
end
That concatenation is the documented recipe. It is not a function in the core because no core function knows Pro exists, and it is not a function here because a host that wants only the Pro half should be able to have it.
Narrow either side with :include:
AuroraMeter.Pro.Telemetry.Metrics.metrics(include: [:exports, :reconciliation])
AuroraMeter.Pro.Telemetry.Metrics exists only when telemetry_metrics is
installed. Without it the module is not defined, every Pro event is still
emitted, and nothing in lib/ references it.
Where next
-
Telemetry (core): the free
half of the catalogue, the tag rules in full, and
redact/2. - The export outbox: the states the gauge counts.
- Reconciliation: what a difference means and what it does not.
- Recovery: what to do about an uncertain outcome.
-
API inventory: every event with a stability class and a
Since.