Telemetry

Aurora Meter Pro emits :telemetry events. It talks to exactly one place, Stripe, through one client seam, and it sends telemetry nowhere: attaching a handler, running a reporter and choosing what leaves your network are your decisions. AuroraMeter.Pro.NoOutboundIoTest asserts that rather than promising it, with an allow list three files long and a reason beside each.

AuroraMeter.Pro.Telemetry.events/0 is this page in machine-readable form. The table below is checked against it, and against lib/, on every mix test: an event that is emitted and not listed here fails the suite, and so does one listed here and not emitted.

The rules for a metric tag are the core’s

They are stated once, in the free core, and Pro uses them: AuroraMeter.Telemetry.tag_allow_list/0, AuroraMeter.Telemetry.forbidden_tags/0 and AuroraMeter.Telemetry.tag_allowed?/2. One rule written down twice is two rules waiting to disagree.

In short: a tag is one of result, kind, exporter, state or worker, plus feature when metrics_feature_label: true. Everything else is correlation metadata. Pro’s own reasons for that, over and above the core’s:

  • tenant_key is on every commercial event and is unbounded by definition.
  • payment_intent_id, item_id, subject_ref and lease_token identify one object each, so a tag on any of them is one time series per object.
  • period_start and watermark are instants. One series per billing period, for ever.
  • attempts is a count. A count is a measurement, not a dimension.

Where a Pro event’s metadata name is bounded but is not an allow-listed name, the shipped preset maps it rather than tagging on it raw: outcome becomes result, subject_kind and action and table become kind, transition becomes state. The name of a dimension is a contract with a reporter, and the library keeps that contract closed so a dashboard built on one Aurora Meter deployment reads the same on another.

AuroraMeter.Telemetry.redact/2 is the supported way to put event metadata in a log line or a span attribute. It drops every identifier here, including payment_intent_id and subject_ref, and turns error into error_class.

Three events under a core-looking name

[:aurora_meter, :alert], [:aurora_meter, :usage_report] and [:aurora_meter, :reconcile] are Pro events in the core’s namespace. A host attaching a handler cannot tell from the name whether the signal is free or commercial, and the core’s own telemetry page does not list them, because the core does not know Pro exists.

They are kept. Renaming them would silently detach every handler already attached to them, which is a worse outcome than an untidy name, and a rename is a 2.0 conversation. Every Pro event added since uses [:aurora_meter, :pro, ...].

The events

Event Measurements Metadata Tags Since
[:aurora_meter, :alert] percent, threshold, usage feature, kind, tenant_key kind 0.1.0
[:aurora_meter, :usage_report] delta feature, period_start, tenant_key none 0.1.0
[:aurora_meter, :reconcile] drift feature, tenant_key none 0.1.0
[:aurora_meter, :pro, :auto_top_up] amount_cents reason, result, tenant_key result 0.3.0
[:aurora_meter, :pro, :outbox, :deliver] duration_ms attempts, outcome, reason, state, subject_kind state 0.4.0
[:aurora_meter, :pro, :outbox, :gauge] accepted, confirmed, leased, legacy_pending, oldest_pending_age_seconds, pending, quarantined, rejected, uncertain none none 0.4.0
[:aurora_meter, :pro, :outbox, :stale_worker] count item_id, lease_token, transition none 0.4.0
[:aurora_meter, :pro, :credits, :gauge] armed, below_threshold, enqueued, reconciliation_required operation none 0.4.0
[:aurora_meter, :pro, :credits, :wallet_wide_fallback] count operation, payment_intent_id, tenant_key none 0.6.0
[:aurora_meter, :pro, :webhook] duration_ms api_version, duplicate, outcome, type none 0.4.0
[:aurora_meter, :pro, :cutover] none action, feature, watermark none 0.4.0
[:aurora_meter, :pro, :correction] count item_id, net_value, original_age_seconds, outcome, subject_ref, window none 0.4.0
[:aurora_meter, :pro, :reconciliation, :difference] expected, observed feature, kind, tenant_key kind 0.4.0
[:aurora_meter, :pro, :reconciliation, :run] confirmed, duration_ms, groups, items_opened, requests stopped none 0.4.0
[:aurora_meter, :pro, :recovery, :action] count action, outcome, subject, tenant_key none 0.4.0
[:aurora_meter, :pro, :retention, :prune] deleted, duration blocked, table none 1.0.0
[:aurora_meter, :pro, :transition] count ref, result, tenant_key, to_plan_id, to_version result 1.0.0
[:aurora_meter, :pro, :usage_report, :plan_fallback] count feature, period_start, plan_id, plan_version, tenant_key none 1.0.0
[:aurora_meter, :pro, :provider] duration, monotonic_time, system_time kind, operation, reason, result, stacktrace result 1.0.0

Alerting, and the provider span

Alerts is six worked examples over this catalogue, each with the signal, the derivation of its threshold from a value you control, a severity and a runbook link, plus a “do not alert on this” section. They are deployment examples and not service level objectives.

[:aurora_meter, :pro, :provider] is a span around one outbound provider call. It is emitted at the seam where Pro calls the configured client, so one span is one HTTP attempt and never the loop that retries it, which is what makes a latency distribution over these spans mean anything. operation is one of AuroraMeter.Pro.Telemetry.provider_operations/0, a closed set of callback names; it is metadata rather than a tag because the tag allow list does not carry it. No request body, no response body, no customer identifier and no payment identifier is on the span.

AuroraMeter.OpenTelemetry (core, optional) turns this event and [:aurora_meter, :pro, :outbox, :deliver] into spans in the host’s own OpenTelemetry SDK, along with the slow half of core’s catalogue. It attaches to the Pro event names as data and references no Pro module.

Failure mode, signal, runbook

Core’s docs/telemetry.md carries the same table for the free half: buffered loss, flush failure, cluster divergence, durable event outcomes, holds, expiry and recurring grants. This one covers the commercial half.

State Signal Runbook
Export backlog growing pro.outbox.gauge.pending and pro.outbox.gauge.oldest_pending_age_seconds rising operations.md section 10
Provider outcome unknown pro.outbox.gauge.uncertain above zero; pro.outbox.deliver with an uncertain outcome recovery.md
Item quarantined by a mapping, mode or window judgement pro.outbox.gauge.quarantined above zero outbox.md
Fenced write from a stale worker [:aurora_meter, :pro, :outbox, :stale_worker] outbox.md
Provider-side difference found [:aurora_meter, :pro, :reconciliation, :difference] reconciliation.md
Reconciliation stopped early pro.reconciliation.run metadata stopped reconciliation.md
Local over-reporting against the provider [:aurora_meter, :reconcile] with a non-zero drift operations.md section 5
Payment outcome unresolved pro.auto_top_up with a failed result; pro.credits.gauge.reconciliation_required above zero recovery.md
Payment could not be attributed to one lot [:aurora_meter, :pro, :credits, :wallet_wide_fallback] top-ups.md
Guarded recovery action performed [:aurora_meter, :pro, :recovery, :action] plus its audit row operations.md section 7
Webhook handling failing pro.webhook with a failing outcome subscriptions.md
Paid transition not confirmed by the provider [:aurora_meter, :pro, :transition] absent past its effective time subscriptions.md
A usage row could not be attributed to a recorded plan version [:aurora_meter, :pro, :usage_report, :plan_fallback] usage-reporting.md
Version 10 upgrade incomplete, deliverer cancelling pro.outbox.gauge.legacy_pending at 1 cutover.md
Retention refused to prune pro.retention.prune with blocked: true retention.md
A scheduled worker stopped running silence from the event that worker emits scheduler.md
Usage alert threshold crossed [:aurora_meter, :alert] operations.md section 3

The backlog gauge, and why absence matters

[:aurora_meter, :pro, :outbox, :gauge] is one event with a measurement per state, not one event per state. A host writes last_value("aurora_meter.pro.outbox.gauge.pending") and gets a clean series per state with no tag at all, which is the cheapest possible answer to the cardinality question.

AuroraMeter.Pro.Outbox.Deliverer emits one at the end of every run. AuroraMeter.Pro.Outbox.gauge/0 emits one on demand, for a host whose queue is paused or that runs the work on another scheduler.

When the aggregate query fails, no sample is emitted at all. Not a zero. A zero on this gauge reads as “the backlog is empty”, and the case it exists for is the one where the backlog is not empty and nobody can see it. The same rule applies to the run itself: a paused deliverer produces no samples, so the last one goes stale and staleness is the signal. Alert on the age of the last sample as well as on its value.

Metrics presets

# mix.exs
{:telemetry_metrics, "~> 1.0"}
defmodule MyApp.Telemetry do
  def metrics do
    AuroraMeter.Telemetry.Metrics.metrics() ++ AuroraMeter.Pro.Telemetry.Metrics.metrics()
  end
end

That concatenation is the documented recipe. It is not a function in the core because no core function knows Pro exists, and it is not a function here because a host that wants only the Pro half should be able to have it.

Narrow either side with :include:

AuroraMeter.Pro.Telemetry.Metrics.metrics(include: [:exports, :reconciliation])

AuroraMeter.Pro.Telemetry.Metrics exists only when telemetry_metrics is installed. Without it the module is not defined, every Pro event is still emitted, and nothing in lib/ references it.

Where next