Retention

Aurora Meter Pro keeps financial history for ever. The acknowledgement ledger, the payment accounts, the alert dedupe rows, the reporting aggregates and the source cutovers are never deleted by anything in this package, at any age, under any option, and nothing prunes anything at all until you schedule it.

Five tables grow with traffic rather than with tenants, and each is genuinely disposable once its own condition holds. AuroraMeter.Pro.Retention is that list.

AuroraMeter.Pro.Retention.plan([])
#=> {:ok, %{audit_events: 412_000, outbox_items: 1_204, webhook_events: 8_910,
#=>         recovery_actions: 3, reconciliation_items: 12}}

AuroraMeter.Pro.Retention.prune([])

plan/1 is a dry run. It runs the prune’s own predicate with count(*) instead of a DELETE and writes nothing at all, so the number it gives you is the number prune/1 then removes, generated from one predicate rather than from two that could drift.

The free core has its own list on the same terms, in the page of the same name in its documentation, and neither package prunes the other’s tables. Schedule both.

The allow list

That is the whole list. There is no option, no configuration key and no argument that adds a table to it. :only narrows the list and can never widen it:

AuroraMeter.Pro.Retention.prune(only: [:audit_events])

AuroraMeter.Pro.Retention.prune(only: [:usage_reports])
#=> ** (ArgumentError) :usage_reports is not a table
#=>    AuroraMeter.Pro.Retention may delete from...

Every window has a floor of one day and a value below it is refused at boot. A retention window is compared against a stored timestamp, and the clocks that write those timestamps step backwards by a few hundred milliseconds to a few seconds; at a day that cannot invert the comparison, and at a minute it can. This decision deletes data that cannot be recovered, so the small value is refused where an operator can still read the message. For a one-off narrower cutoff, pass :older_than_days to a single call.

What is never deleted, and why each one

aurora_meter_usage_reports is the acknowledgement ledger. last_reported per tenant, feature and period is what makes the next delta correct. Delete an old row and the reporter believes that period was never reported and re-sends the whole period’s usage, which is a double bill. Never prunable, at any age.

aurora_meter_credit_accounts is the provider identity of a paying customer and the state of any top-up attempt in flight.

aurora_meter_alerts is a dedupe table: the row is written before the alert is sent and removed only when delivery failed, so deleting an old row re-alerts a customer about a threshold they crossed a year ago. Prunable in principle once the period is long past, and deliberately not: the saving is one row per tenant, feature, period and threshold, and the risk is a customer email.

aurora_meter_rollups is the reporting aggregate the history charts read. Rebuilding it is AuroraMeter.Pro.Rollup‘s job and deleting it is nobody’s.

aurora_meter_source_cutovers holds one row per feature naming the period at which that feature moved from buffered counting to durable events. It is one row per feature, for ever: it does not grow with traffic, and both the reporter and the reconciler read it to decide which source a past period should be billed from. Deleting one would change the commercial answer for every period after it.

An outbox item that is not confirmed is never deleted whatever its age. pending and leased are in flight, accepted and uncertain are waiting on the provider, and rejected and quarantined are records a person still has to look at. An open reconciliation item is never deleted for the same reason.

A test in this package derives the tables the migrations create and fails unless every one appears in exactly one of the two lists, in both directions. A table nobody classified is a table that grows for ever, or worse, one a later prune reaches.

Why 90 days for a confirmed outbox item

A confirmed item is a settled fact: the provider accepted it and a later reconciliation cycle found a summary that accounts for it. The evidence that it happened does not live only in that row. aurora_meter_usage_reports keeps the acknowledged position for that tenant, feature and period for ever, and the provider keeps its own record. Ninety days is comfortably longer than the reconciler’s lookback, so nothing this package does reads a confirmed item older than that.

If you would rather keep them, set outbox_retention_days higher, or leave the table out of the schedule with :only. Nothing breaks either way; the table grows.

Scheduling

config :my_app, Oban,
  queues: [aurora_meter: 5],
  plugins: [{Oban.Plugins.Cron, crontab: [
    {"30 3 * * *", AuroraMeter.Pro.Retention.Pruner}
  ]}]

AuroraMeter.Pro.cron_entries/1 includes it. "30 3 * * *" is the schedule AuroraMeter.Pro.AuditLog.Pruner documented before this worker existed, so a host moving the entry across keeps its existing time.

AuroraMeter.Pro.AuditLog.Pruner is deprecated and still works: it prunes the audit log and only the audit log, exactly as it did. Move the crontab entry, and do not register both, because both sweep the audit log.

Pausing

One operation per table:

AuroraMeter.Operations.pause("pro_retention:outbox_items")
AuroraMeter.Operations.resume("pro_retention:outbox_items")

A paused table is reported as a :paused reason rather than as a failure, and the other tables are still pruned.

When one run cannot keep up

prune/1 removes at most :max_items rows per table per call (default 50,000), so it cannot run for an unbounded time. A run that hits that bound with rows still eligible says so:

AuroraMeter.Pro.Retention.prune([])
#=> {:blocked, %{audit_events: 50_000, ...},
#=>  [%{table: :audit_events, reason: :budget_exhausted,
#=>     detail: %{deleted: 50_000, max_items: 50_000, batch_size: 1000}}]}

That is not an error. It is the signal that the table is growing faster than one nightly run removes, and the two answers are to raise :max_items or to run the worker more often. plan/1 tells you how much is waiting.

The audit log is usually the one that needs this: it is one row per API request, so a busy installation can add more rows in a day than a default run removes.

Archiving

Aurora Meter Pro uploads nothing anywhere and ships no archiver. Building one would be a second storage system, and you already have a better one: your backups.

If you want a copy of what a prune is about to remove, take it with COPY before the prune:

COPY (SELECT * FROM aurora_meter_audit_events
       WHERE inserted_at < now() - interval '30 days')
  TO '/var/backups/aurora_audit_events.csv' WITH CSV HEADER;

For the tables that are never pruned, which is where your commercial history lives, the archive is your ordinary database backup and nothing here changes that. If you want them out of the hot database for size reasons, that is a partitioning or a warehouse question rather than a retention one: a usage report or a source cutover that is gone cannot be reconciled against the provider afterwards.