Metering API Usage in Phoenix in Real Time with ETS

Phoenix usage metering usually starts as one line in a plug: Repo.update_all(from(u in Usage, where: ...), inc: [count: 1]). It is correct, it is atomic, and it is on the hot path of every request your customers make. At a few hundred requests a second it is a noticeable share of your connection pool. At a few thousand it is the reason p99 latency doubled.

This post is about moving that increment to ETS and keeping everything else you had: durable totals in Postgres, a usage bar that updates live in the browser, and numbers that stay correct across a deploy and across a cluster. It is how the free MIT core of Aurora Meter works, and you can build the same thing by hand if you would rather.

Why the increment belongs in ETS

Three options exist on the BEAM for a counter that many processes bump concurrently.

A row in Postgres. Atomic, durable, and expensive: a pooled connection, a network round trip, a row lock and a WAL write per request. It also serialises every increment for one tenant on one row lock, so a busy customer slows their own requests down.

A GenServer per tenant. Cheap, but a process mailbox is a queue. Every increment for a tenant waits behind every other increment for that tenant, a crash loses the state, and you now have a registry, supervision and a cleanup strategy to write.

ETS with :ets.update_counter/4. Atomic, executed inside the ETS table without a process, and lock-free for distinct keys. Thousands of processes can increment thousands of keys with no coordination. A single hot key serialises on the table’s write lock, which still runs at tens of thousands of increments a second.

@table :aurora_meter_counters

def track(tenant, feature, quantity \\ 1) do
  key = {to_key(tenant), feature, current_period()}
  :ets.update_counter(@table, key, {2, quantity}, {key, 0})
end

The fourth argument is the default row, so a counter that has never been seen springs into existence at zero on first use. That single call is the entire write path. Measured with eight processes over distinct counters, Aurora Meter records roughly 5.5 million increments a second on a laptop; with all eight hammering one key, about 53 thousand. Both numbers are in the published benchmark.

Getting it into Postgres

ETS is memory. A deploy, a crash or a node restart empties it. You need two things: a flusher that persists the counters on an interval and on clean shutdown, and a seed that repopulates a cold counter from the last flushed value the first time it is read.

The flusher is a GenServer that wakes every few seconds, walks the table for keys touched since the last flush, and upserts the values. Trap exits so a normal shutdown flushes one last time:

def init(opts) do
  Process.flag(:trap_exit, true)
  schedule_flush(opts)
  {:ok, opts}
end

def terminate(_reason, state), do: flush(state)

Only the flusher touches the database on the write side. The read side touches it exactly once per counter per node, when usage/2 finds no ETS row and seeds it from Postgres. After that every read is an ETS lookup.

Be explicit about what this buys and costs. A buffered counter can lose at most one flush interval of increments in a hard crash. For a usage bar or a soft quota that is fine. For anything that appears on an invoice, mark the feature durable so each increment also writes a raw event row synchronously, and let the buffered counter be the fast view of the same data:

config :aurora_meter, durable_features: [:ai_generations]

Fan-out to LiveView

A usage meter that updates when the customer refreshes is a usage meter nobody trusts. Because the flusher already tracks which keys changed, a broadcaster can publish those values over Phoenix.PubSub on a shorter interval, typically once a second. A LiveView subscribes on mount and re-renders on each message; there is no polling and no database read anywhere in the loop.

def mount(_params, _session, socket) do
  if connected?(socket), do: AuroraMeter.LiveView.subscribe(socket.assigns.org)
  {:ok, socket}
end

<.usage_meter tenant={@org} feature={:ai_generations} />

The component renders the used, limit and percent values from the live counter and updates within a second of any increment on any node. LiveView and Phoenix.HTML are optional dependencies of the core, so a headless service can meter without them.

Periods without a reset job

Usage resets. The simplest correct approach is to make the period part of the key, as the current_period() call above does. When the UTC month rolls over, the next increment lands on a fresh key and the old one stays behind for history. No cron job, no race between the reset and in-flight increments, nothing to forget on a deploy. The flusher also records day buckets, so AuroraMeter.history(org, :ai_generations, days: 30) returns a daily series for charts.

Once billing periods follow a subscription rather than the calendar, the period source becomes pluggable. That is one of the things Aurora Meter Pro provides: periods aligned to the Stripe subscription’s current period, synced by webhook. The usage-based billing guide covers why that matters for invoices.

Making it correct across a cluster

Every node runs its own ETS table, so out of the box two nodes each see their own increments. Two strategies are wrong. Having every node write its absolute value to the same row means the last flusher wins and the total is undercounted. Routing all increments to one node reintroduces the single mailbox you left Postgres to avoid.

The strategy that works is deltas. Each node flushes the difference since its last flush (value = value + delta), so the Postgres row is the cluster total. Between flushes, nodes exchange their deltas over the same distributed PubSub LiveView already uses, and each node re-bases its local view on the persisted total after every flush. A value read on any node is then the true total minus at most the other nodes’ last second of increments.

The consequence for hard limits is worth stating plainly: a burst spread across N nodes can overshoot a cap by whatever the other N minus 1 nodes admitted within one broadcast interval. For most products that is a handful of requests once a month; for a limit that must be exact, enforce it on a single node or in the database. Aurora Meter 0.3 ships this delta model, and the clustering guide spells out the timing.

Observability

A meter you cannot observe is a meter you will debug in production. Emit telemetry from the three moving parts: the track call (tenant, feature, quantity), the reservation (outcome: admitted or refused), and the flush (rows written, duration). With Telemetry.Metrics those become counters and distributions in LiveDashboard, and a sudden drop in flush counts is the earliest sign that a node stopped persisting.

:telemetry.attach("log-flushes", [:aurora_meter, :flush], fn _event, m, _meta, _ ->
  Logger.info("flushed #{m.rows} counters in #{m.duration}ms")
end, nil)

Doing it in one dependency

Everything above is a few hundred lines, plus tests for the concurrency edges, plus the cluster case, plus the shutdown flush you will forget the first time. Aurora Meter’s free core is exactly that code, MIT licensed, with a compile-time plan DSL and the atomic with_quota/4 gate on top:

# mix.exs
{:aurora_meter, "~> 0.3"}

mix aurora_meter.gen.migration -r MyApp.Repo
mix ecto.migrate

# application.ex, after the Repo and PubSub
children = [MyApp.Repo, {Phoenix.PubSub, name: MyApp.PubSub}, AuroraMeter, MyAppWeb.Endpoint]

Then AuroraMeter.track(org, :api_requests) in your API plug and <.usage_meter /> in your dashboard. The docs cover configuration, durable features, clustering and telemetry, and the entitlements post shows how plan limits sit on top of the counters. When it is time to invoice the overage, Aurora Meter Pro adds Stripe Checkout, webhook sync and the usage reporter without changing a line of the metering code. If you want a metered API product with all of this already built, the Aurora API Starter ships one on this core.