Ship it: the production checklist

Everything so far has been about making it work. This guide is about it staying correct when nobody is looking: on several servers, through a crash, through Stripe having a bad afternoon, and six months from now when somebody “simplifies” a function they do not understand.

Work down it before the first real card.

#1. Running on more than one server

Every node counts into its own memory. Nobody coordinates, because coordinating is what would make counting slow.

Instead, each node writes deltas rather than totals, so nodes add up rather than overwriting each other, and the row in Postgres is the cluster total. On top of that, nodes gossip their deltas over PubSub every second and re-base on the stored total every five seconds.

  node A  --+-- gossip every 1s --+--  node B
            |                     |
            +----> Postgres <-----+     deltas, every 5s
                    (the total)

So a value read on any node is the true total minus, at worst, the other nodes’ last second of counting.

What you need for this: the distributed PubSub you already run for LiveView. On a single node nothing changes and there is nothing to configure.

A hard limit is enforced against the local view. A burst arriving on several nodes at once can overshoot a cap by whatever the other nodes admitted in that last second. For a quota that is a product decision (“50 a month”) that is fine. For something that must be exact to the unit under a coordinated burst, that is a database constraint’s job, not a meter’s.

#2. Decide what may be lost, and write it down

Counters live in memory and flush to Postgres every five seconds and on a clean shutdown. A crash can lose up to five seconds of counts, including anything backed up during a database outage. The flusher keeps its pending batch across its own restart.

That is the correct trade for most things and the wrong trade for some. Choose per feature:

config :aurora_meter, durable_features: [:contract_reviews]

A durable feature writes an event row on every call. Correct, slower, a database write per request.

The checklist item is not “pick one”. It is write down which features are durable and why, in a comment next to the config, so the next person does not quietly add the expensive one to the fast list.

#3. The four alarms

Not dashboards. Alarms, that wake somebody.

Signal	Why it matters
[:aurora_meter, :flush, :error], repeatedly	counts are piling up in memory and will be lost on restart
AuroraMeter.Pro.Reconcile.run/0 returning anything	you have told Stripe about more usage than you have recorded
[:aurora_meter, :credits, :settle] with overrun: true, often	your cost estimates are wrong and customers are going negative
automatic recharge disabled for many customers at once	this is Stripe being unwell, not a lot of bad cards

One failed flush is noise. A stream of them is the only warning you get before usage disappears.

:telemetry.attach(
  "overruns",
  [:aurora_meter, :credits, :settle],
  fn
    _event, %{amount: amount}, %{overrun: true, tenant_key: key, reference: ref}, _cfg ->
      Logger.warning("overrun #{ref} for #{key}: #{amount}")

    _event, _measure, _meta, _cfg ->
      :ok
  end,
  nil
)

And the fifth signal, which has no telemetry event because only you know how long your work takes: open holds older than your longest job. Those are reservations nothing will ever close, and money nobody can spend.

defmodule MyApp.Health.Holds do
  def stale_count do
    DateTime.utc_now()
    |> DateTime.add(-6 * 3600, :second)
    |> then(&AuroraMeter.Credits.pending_holds(older_than: &1))
    |> length()
  end
end

#4. A health check that asks whether the books balance

defmodule MyApp.Health.Billing do
  @moduledoc "Invariants that should be true at all times."

  import Ecto.Query

  def check do
    %{
      drift: AuroraMeter.Pro.Reconcile.run(),
      stale_holds: MyApp.Health.Holds.stale_count(),
      negative_balances: negative_balances()
    }
  end

  defp negative_balances do
    MyApp.Repo.aggregate(
      from(b in "aurora_meter_credit_balances", where: b.balance < 0),
      :count
    )
  end
end

Every one of those should be zero or empty.

Checking it once by hand proves very little. Checking it every five minutes and alerting on a change turns “it was correct when I looked” into “I will know within minutes when it stops being correct”. For a system that moves money, that is worth more than another round of code review.

#5. Scheduled work

config :my_app, Oban,
  queues: [aurora_meter: 5],
  plugins: [
    {Oban.Plugins.Cron,
     crontab: [
       {"*/5 * * * *", AuroraMeter.Pro.UsageReporter},
       {"*/10 * * * *", AuroraMeter.Pro.Alerts},
       {"15 2 * * *", AuroraMeter.Pro.Rollup},
       {"*/30 * * * *", AuroraMeter.Pro.Credits.Expirer},
       {"*/5 * * * *", AuroraMeter.Pro.Credits.AutoTopUpSweeper},
       {"30 3 * * *", AuroraMeter.Pro.AuditLog.Pruner}
     ]}
  ]

• UsageReporter sends metered usage to Stripe. Without it, overage is counted and never billed.
• Alerts fires your threshold callbacks.
• Rollup builds the daily and monthly aggregates the history charts read, instead of scanning the event table.
• Expirer expires promotional credit that has passed its date.
• AutoTopUpSweeper catches accounts sitting below their threshold that the crossing hook missed.
• Pruner deletes old audit rows in bounded batches, so it does not lock the table.

Only schedule what you use. A product with no prepaid credit does not need the last three; a product with no metered features does not need the first.

Every one of these runs under a database advisory lock and Oban’s uniqueness, so several servers running the same crontab do not trip over each other. They must all point at the same database.

#6. The audit log

One row per API call: who, what route, what status, how long, and what it cost.

# router.ex
pipeline :api do
  plug :accepts, ["json"]
  plug MyAppWeb.Authenticate

  plug AuroraMeter.Pro.Plugs.AuditLog,
    tenant: &MyAppWeb.Audit.tenant/1,
    actor: &MyAppWeb.Audit.actor/1,
    metadata: &MyAppWeb.Audit.metadata/1
end

defmodule MyAppWeb.Audit do
  def tenant(conn), do: conn.assigns[:current_org]
  def actor(conn), do: conn.assigns[:current_user] && "user:#{conn.assigns.current_user.id}"
  def metadata(conn), do: %{"api_key_id" => conn.assigns[:api_key_id]}
end

Everything shaped like your app arrives as a function of the conn, so the package never guesses at your schema.

A tenant resolver returning nil skips the write. That is how unauthenticated traffic stays out of the log, so put the plug after your authentication, never before.

The plug records on register_before_send/2, so it never delays the response. A resolver that raises, or a write that fails, is a warning in the log and never an error in the request being audited.

Reading it back:

AuroraMeter.Pro.AuditLog.list(org, status_class: :server_error, limit: 50)
# %{data: [%AuroraMeter.Pro.Schema.AuditEvent{}, ...], has_more: true,
#   next_cursor: "..."}

AuroraMeter.Pro.AuditLog.stats(org, from: DateTime.add(DateTime.utc_now(), -86_400))
# %{total: 1_284, by_status_class: %{success: 1_270, client_error: 12,
#   server_error: 2}, p50_ms: 180, p95_ms: 940}

Paging is a keyset cursor, so it stays stable while new rows arrive underneath it.

config :aurora_meter_pro,
  audit_log_retention_days: 30,
  audit_log_writer: :async     # :sync in tests

The async writer needs a task supervisor in your tree:

children = [
  {Task.Supervisor, name: AuroraMeter.Pro.TaskSupervisor},
  # ...
]

#7. Tests that can actually fail

Most of the bugs in this library’s own history were tests that could not fail. Four worth having in your suite:

A cap holds under concurrency. Twenty tasks, a cap of two, exactly two successes. This is the only test that fails if somebody replaces with_quota/4 with a check followed by a track.

A redelivered webhook does not pay twice. Handle the same event object twice, assert the balance moved once. Stripe retries any non-2xx for days, so this is not an edge case.

A cancellation revokes the plan. Sync an active subscription, assert the paid plan; sync a cancelled one, assert the default plan.

The ledger inside your own transaction. And write this one without the Ecto SQL sandbox, in a test that uses a real transaction. The sandbox holds a transaction of its own, so yours is nested inside it and an abort unwinds no further than the sandbox’s savepoint. This whole class of bug is invisible in a sandboxed DataCase.

Helpers that make all four easy:

AuroraMeter.Test.reset!()             # empty the in-memory counters
AuroraMeter.Test.flush!()             # force the pending numbers to Postgres
AuroraMeter.Test.broadcast!()         # push a live update now, do not sleep
AuroraMeter.Test.unique_tenant("org") # a fresh key, so async tests do not collide
AuroraMeter.Test.fund!(org, 25_000_000)
AuroraMeter.Test.credit_balance(org)

And keep the network out entirely:

# config/test.exs
config :aurora_meter_pro,
  credits_stripe_client: AuroraMeter.Pro.Credits.StripeClient.Fake,
  audit_log_writer: :sync

#8. Upgrading the packages

Both packages carry a schema version, and a release states which one it needs. Generate a migration for the versions you have not applied yet:

mix aurora_meter.gen.migration -r MyApp.Repo --from 3
mix aurora_meter_pro.gen.migration -r MyApp.Repo --from 3
mix ecto.migrate

Apply migrations before starting the updated application, not after. The changelogs say which version each release needs:

• Aurora Meter changelog
• Aurora Meter Pro changelog

#The checklist itself

Copy this into your own issue tracker.

Configuration

• default_plan is set, and it is your free tier.
• Every feature you sell appears on every plan, so a typo is loud.
• durable_features is decided, with a comment saying why.
• period_source: AuroraMeter.Pro.Period is set if you sell subscriptions.

Stripe

• The webhook plug is mounted before Plug.Parsers.
• The endpoint is subscribed to the subscription events, the payment events, and the refund and dispute events.
• Every metered price has a graduated tier whose free allowance equals your plan’s included:.
• You have run a real checkout in test mode and watched the plan change.
• You have refunded a test payment and watched the balance come back down.
• You have seen a test-clock invoice with the right total on it.

Operations

• The Oban crontab has only the workers you use, and you have watched each run once.
• Alarms exist for flush errors, reconciliation drift, settlement overruns and mass recharge failures.
• A health check runs every few minutes and alerts on a change.
• The audit log plug is after authentication, and the pruner is scheduled.

Tests

• A concurrency test on your tightest cap.
• A redelivered webhook test.
• A cancellation test.
• Ledger transaction tests that do not run inside the sandbox.

#What to read next

• Build a metered SaaS, end to end if you arrived here first and want the whole thing in order.
• The package READMEs, which are the exact files shipped inside the packages: Aurora Meter and Aurora Meter Pro.