Running it: the scheduled work, and stopping it

Everything in Aurora Meter that happens on a schedule is optional, is an Oban worker, and does work that is safe to do twice. This guide is what each one does, how to turn them on, how to stop one at three in the morning without a deploy, and what to do when something is stuck.

All of it is in the free core. Oban is an optional dependency: without it these modules are not compiled at all.

#The six workers

config :my_app, Oban,
  repo: MyApp.Repo,
  queues: [aurora_meter: 5],
  plugins: [{Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()}]

cron_entries/0 returns the recommended schedule, so the list below is not something you maintain by hand:

They all use one queue, :aurora_meter, shared with Pro.

Then check the wiring when your application starts:

def start(_type, _args) do
  AuroraMeter.Oban.validate!(otp_app: :my_app)
  Supervisor.start_link(children, opts)
end

validate!/1 raises AuroraMeter.Oban.ConfigError listing every problem it found in one pass, rather than one per restart. A missing queue and a worker scheduled twice come back together.

#Nothing here is promised a single run, and every job is built for that

A cron plugin can tick twice across a leader change. A rescued job runs again. Two nodes can both be told to sweep. Aurora Meter does not assume otherwise.

What each operation gives you instead is that a second run finds the work already done and says so:

• expiry re-reads the grant’s expired_at under that row’s own lock and refuses when it is already set, counting the attempt as skipped
• a hold is settled or released through a re-read of status = 'pending' under the hold row’s lock, so the loser is told :already_closed
• a plan transition’s every effect is an update conditional on the transition still being pending
• a delete is a delete

The unique option on each worker is defence in depth, not the guarantee. Removing it would waste work. It would not move money.

The same thinking applies to your own code. A crash between a batch committing and its checkpoint being written re-runs that batch, and that is safe only because the effects are idempotent. Write yours the same way.

#Changing the schedule, or leaving a worker out

cron_entries/1 takes options rather than asking you to copy the list:

AuroraMeter.Oban.cron_entries(
  exclude: [AuroraMeter.Oban.HoldReconciliation],
  schedules: %{credit_expiry: "0 4 * * *"}
)

:include keeps only what you name, :exclude drops what you name, and :schedules replaces a worker’s cron expression. Naming a worker in :schedules is also the only way to schedule one that has no default, which is how you would put EventsReplay on a timer if you ever wanted to.

An entry appears only when the operation behind it is actually compiled into your build. A worker whose operation is missing cancels with {:cancel, :not_implemented} rather than failing every five minutes for ever.

#Pausing without a deploy

Rescheduling is a deploy. Pausing is a function call, and it is the one you want at three in the morning.

AuroraMeter.Operations.pause("credit_expiry:global")
AuroraMeter.Operations.paused?("credit_expiry:global")   # true
AuroraMeter.Operations.resume("credit_expiry:global")

A paused operation’s next job cancels rather than running, and the pause is read before every batch, including the first, so pausing mid sweep stops it at the next batch boundary rather than at the end.

Resuming starts from the checkpoint, not from the beginning:

AuroraMeter.Operations.checkpoint("credit_expiry:global")
# %{name: "credit_expiry:global", cursor: %{...}, counts: %{...},
#   state: "paused", updated_at: ~U[...]}

AuroraMeter.Operations.list()

The names, because they are strings and a typo raises:

credit_expiry:global
hold_reconciliation:global
credits_recurrences:global
retention:flush_receipts
retention:replay_checkpoints

AuroraMeter.Oban.PlanTransitions has no checkpoint and cannot be paused. Its batch loop lives inside one job and carries its cursor only for the length of that run. To stop it you drop its crontab entry and deploy, or pause the queue in Oban. This is the one exception to the table above.

clear_checkpoint/1 deletes the row, and clearing a checkpoint also clears a pause, which is worth knowing before you use it to “reset” something.

#Credit holds that nobody closed

A hold is opened before work and settled after it. A process that dies in between leaves the hold open and the customer’s money reserved, and no amount of retrying your job will close it, because your job is gone.

The sweep asks a module you write what to do:

config :aurora_meter, credits_hold_reconciler: Notewell.HoldPolicy

defmodule Notewell.HoldPolicy do
  @behaviour AuroraMeter.Credits.HoldReconciler

  @impl true
  def decide(%{reference: "gen:" <> id, age_seconds: age}) when age > 3_600 do
    case Notewell.Generations.get(id) do
      %{state: :done, cost: cost} -> {:settle, cost}
      %{state: :failed} -> :release
      nil -> :release
    end
  end

  def decide(_hold), do: :keep
end

The hold you are given carries tenant_key, reference, amount, held_at, age_seconds and metadata, and the three decisions are :keep, :release and {:settle, amount}.

Four rules about your callback, and they are all in your favour:

• It may be called more than once for the same hold. Make it a question about your own records, not a state machine.
• It runs outside any transaction, in its own process, with no row locked. Doing I/O in it is fine.
• Anything that is not a valid decision means :keep: a raise, an exit, a timeout past credits_hold_reconciler_timeout, a value that is not one of the three, or no reconciler configured at all. There is no path on which a failure releases money.
• Without a reconciler, every hold is kept. That is the default, and it is the right default.

Dry run it before you trust it:

AuroraMeter.Credits.reconcile_holds(older_than: 3_600, reconciler: fn _ -> :keep end)

The report counts examined, kept, released, settled, already_closed and failed.

#Retention

Two tables are disposable, and they are the only two the pruner will ever touch:

Events, event totals, the credit ledger, subscriptions, plan versions, plan transitions and every sweep cursor are never deleted at any age under any option. The allow list is compiled in, and :only can narrow it but not widen it.

AuroraMeter.Retention.plan(older_than: 30)
# {:ok, %{flush_receipts: 12_400}}      a dry run; writes nothing

AuroraMeter.Retention.status()
# %{cutoff: ..., heartbeats: [...], versions: [...]}

AuroraMeter.Retention.prune(only: [:flush_receipts], batch_size: 1_000)

A receipt is only deletable when every node’s flush heartbeat proves no node still holds an older batch, so the order of operations on a new install matters: deploy, let every node write a heartbeat, confirm with status/0, and only then add the crontab entry.

{:blocked, report, reasons} is a normal outcome and not a failure. The worker maps it to a successful job, because “I did not delete anything and here is why” is the pruner working.

#Rebuilding a projection

If a projection ever disagrees with the event log, the events are the truth and the projection is rebuildable:

AuroraMeter.Events.Replay.status()

AuroraMeter.Events.Replay.run(batch_size: 5_000, compare: :require_match, activate: true)
# {:ok, %{generation: 4, scanned: 1_204_000, projected: 1_204_000,
#         differences: 0, activated: true, duration_ms: 41_233}}

compare: :require_match is the default and refuses to activate a generation that does not agree with the one in use, answering {:error, {:projection_mismatch, detail}}. compare: :report builds it and tells you the differences without activating, which is what you want the first time. Runs resume from their checkpoint, and a second concurrent run is {:error, {:already_running, status}} rather than a race.

prune/1 removes an old generation once you are sure. It refuses to remove the active one.

#What to watch

Alarms, not dashboards. The two that matter most:

• [:aurora_meter, :flush, :error] repeatedly. Counts are piling up in memory and a restart takes them.
• [:aurora_meter, :operations, :batch] with result: :error, repeatedly, for one operation name. Something is stuck and the cursor is not moving.

AuroraMeter.Operations.list/0 is the two line answer to “is anything paused that nobody meant to leave paused”, and it is worth a page in your admin area.

#What to read next

• Ship it: the production checklist is the list to work through before the first real customer.
• Sell credit up front, spend it per request explains holds, which is what the reconciler is cleaning up after.