Phoenix Cluster Usage Metering: Counting Across Nodes Without Double Counting

Metering usage in a single Phoenix node is a solved problem. One ETS table, :ets.update_counter/4 on the request path, a flusher that writes the totals to Postgres every few seconds, and you have a counter that costs nothing and never blocks. The ETS metering post walks through exactly that build.

Then you run fly scale count 3, or your Kubernetes deployment scales to four replicas, and the number breaks. Not loudly, which is the problem. The dashboard shows a plausible figure, the invoice shows a different plausible figure, and a month later a customer emails to ask why they were billed for 41,000 API calls when their own logs say 58,000.

Phoenix cluster usage metering is the fix: a counting model where every node meters locally, nobody blocks on anybody else, and the number in Postgres is the total for the whole fleet rather than whatever the last node to write happened to remember. This post covers the model, the code, the two failure modes it still has, and the honest limits of enforcing a hard cap across nodes.

The three answers that do not work

Absolute writes. Each node keeps its own count and writes it to the same row: UPDATE usage SET value = 1_402 WHERE .... Node A has counted 1,402, node B has counted 900, and whichever flushes last wins. The row says 900 or 1,402, never 2,302. This is the default outcome if you take a single-node design and simply run more copies of it, and it silently undercounts, which means you silently undercharge.

One counting process. Route every increment to a single named process with :global or a singleton node. Now the count is correct and you have rebuilt the thing you left Postgres to escape: one mailbox that every request in the cluster queues behind, plus a cross-node message per increment, plus a single point of failure whose restart loses the buffer.

Redis. INCR is atomic and cluster-wide, so this works. It also adds a network round trip to every metered request, an operational dependency to run and pay for, and a second source of truth that has to be reconciled with the Postgres data your invoices are built from. On the BEAM you can usually skip it.

The fourth option, incrementing a Postgres row per request, is correct and is the reason you started looking at ETS. At a hundred requests a second it is a row lock on the hot path, and at a few thousand it is the reason your p99 doubled.

The model that works: local counters, delta flushes, gossiped reads

Three rules, and they fit together:

1. Every node counts into its own ETS table. The write path never leaves the node.
2. Every node flushes deltas, not absolutes. The database statement is value = value + Δ, where Δ is what this node has counted since its last successful flush. Nodes add up instead of overwriting each other, so the Postgres row is the cluster total.
3. Nodes gossip their unflushed deltas over PubSub, and re-base on the persisted total after each flush. A read on any node is then the true total minus at most what the other nodes have counted since their last broadcast.

The write path stays a single atomic ETS operation:

defmodule MyApp.Meter.Store do
  @table :usage_counters

  # {key, value, flushed}
  def bump(tenant_id, feature, n \\ 1) do
    key = {tenant_id, feature, period()}
    :ets.update_counter(@table, key, {2, n}, {key, 0, 0})
  end

  def local_delta(key) do
    case :ets.lookup(@table, key) do
      [{^key, value, flushed}] -> value - flushed
      [] -> 0
    end
  end

  defp period, do: Date.utc_today() |> Date.beginning_of_month()
end

update_counter/4 is lock-free and is not serialised through a process mailbox, so eight processes on the same node bump the same counter concurrently without queueing. The third element in the tuple records how much of the value has already been persisted, which is what makes a delta computable without a second table.

The flusher takes the difference, writes it as an increment, and advances the flushed marker:

defmodule MyApp.Meter.Flusher do
  use GenServer
  import Ecto.Query
  alias MyApp.{Repo, Meter.Usage}

  @interval :timer.seconds(5)

  def init(_), do: {:ok, schedule()}

  def handle_info(:flush, state) do
    flush()
    {:noreply, schedule(state)}
  end

  # also flush on the way out, so a clean shutdown loses nothing
  def terminate(_reason, _state), do: flush()

  def flush do
    rows =
      for {{tenant_id, feature, period} = key, value, flushed} <- :ets.tab2list(:usage_counters),
          value > flushed do
        %{tenant_id: tenant_id, feature: feature, period: period, value: value - flushed, key: key}
      end

    if rows != [] do
      Repo.transaction(fn ->
        Repo.insert_all(
          Usage,
          Enum.map(rows, &Map.drop(&1, [:key])),
          on_conflict: from(u in Usage, update: [inc: [value: fragment("EXCLUDED.value")]]),
          conflict_target: [:tenant_id, :feature, :period]
        )
      end)

      for %{key: key, value: delta} <- rows do
        :ets.update_counter(:usage_counters, key, {3, delta})
      end
    end
  end

  defp schedule(state \\ nil) do
    Process.send_after(self(), :flush, @interval)
    state
  end
end

on_conflict with inc: is the whole trick. The generated SQL is value = usage.value + EXCLUDED.value, so three nodes flushing 400, 380 and 622 leave a row holding 1,402, in any order, with no coordination between them.

The two failure modes this still has

Be clear-eyed about these, because they are the difference between a dashboard and an invoice.

A crash between the commit and the marker. The transaction commits, the node dies before :ets.update_counter(..., {3, delta}) runs, and the next flush sends the same delta again. Now you have double counted, which is worse than undercounting: you have overcharged somebody.

The fix is an idempotent batch. Give each flush a batch id, and commit the deltas and a receipt row carrying that id in one transaction. On retry, the receipt is already there, so the batch applies nothing and returns the current totals instead.

def flush_batch(batch_id, rows) do
  Repo.transaction(fn ->
    receipt = %{id: batch_id, inserted_at: DateTime.utc_now()}

    case Repo.insert_all(FlushReceipt, [receipt], on_conflict: :nothing) do
      {1, _} -> apply_deltas(rows)
      {0, _} -> :already_applied
    end
  end)
end

Aurora Meter’s free core does this since 0.4.0 (schema version 6 is the receipts table). Committing the batch and its receipt atomically is why retrying an uncertain commit cannot count the same usage twice, even with several nodes writing at once.

Usage lost since the last flush. If the VM is killed rather than shut down cleanly, everything counted since the last successful flush is gone, along with any backlog accumulated during a database outage. For a usage bar this is fine. For anything you invoice it is not, and the answer is to write a durable event row synchronously for those features and accept the write cost on that path only. In Aurora Meter that is one config line:

config :aurora_meter, durable_features: [:ai_generations]

Pick per feature. :api_requests at ten thousand a minute can be buffered. :ai_generations at two dollars each cannot.

How stale is a “live” number?

Between flushes, nodes broadcast their unflushed deltas on a distributed Phoenix.PubSub topic, the same PubSub that LiveView already uses, so there is nothing new to run:

def broadcast_deltas do
  deltas = for {key, value, flushed} <- :ets.tab2list(:usage_counters), do: {key, value - flushed}
  Phoenix.PubSub.broadcast(MyApp.PubSub, "meter:deltas", {:deltas, node(), deltas})
end

def handle_info({:deltas, from, deltas}, state) when from != node() do
  {:noreply, put_in(state.peers[from], Map.new(deltas))}
end

A read is then the persisted total, plus this node’s unflushed delta, plus the last known delta from each peer. With a one second broadcast interval and a five second flush interval, a value read on any node is the true total minus at most the other nodes’ last second of increments. That is a bounded, explainable staleness, and it is the number to put in the docs your support team reads.

Two practical notes. Ignore your own broadcasts, or you will count your local delta twice. And drop a peer’s entry when its node goes down (:net_kernel.monitor_nodes/1), otherwise a dead node’s delta lingers in every reader’s memory until the next flush re-bases the total.

Enforcing a hard cap: the overshoot you are buying

Here is the part most write-ups skip. If the check is usage < limit against the local view, a burst spread across N nodes can exceed the cap by whatever the other N minus 1 nodes admitted within one broadcast interval. With four nodes, a one second interval and fifty admissions per second per node, the worst case overshoot is about one hundred and fifty units. In practice the window only matters when a customer is already at their limit and traffic is spread evenly, which is rarer than the arithmetic suggests.

So decide per limit what drift costs:

For the first two, the local view with gossip is the right trade: the hot path never blocks and the error is bounded. For the third, do not meter it at all. A seat count is a row count, so enforce it with a unique index and a transaction, where the database gives you exactness for free. The rate limits versus quotas post goes further into which of your limits is which.

Whatever you choose, the gate and the increment must be one operation. Two steps (check, then increment) is a race on a single node, let alone across four:

AuroraMeter.with_quota(org, :ai_generations, fn ->
  generate_report()
end)
# => {:ok, result} | {:error, :limit_exceeded} | {:error, :not_entitled}

Rolling deploys, which is where counters usually break

A deploy replaces nodes one at a time, so the model has to survive a node leaving mid-period:

• Flush on shutdown. A terminate/2 that flushes turns a graceful stop into zero lost usage. Make sure the flusher sits under a supervisor that is stopped before the Repo, or the last write has nowhere to go.
• Seed cold counters from Postgres. A fresh node starting mid-month has an empty table. The first read of a counter must seed it from the last flushed value, or your dashboard shows a customer back at zero on the first of nothing.
• Keep deltas, not absolutes, in the gossip. A new node that broadcasts an absolute value would corrupt every peer’s view. Deltas from a cold node are simply zero until it counts something.
• Order the supervision tree. Repo, then PubSub, then the meter, then the Endpoint. The meter needs both dependencies before the first request can arrive.

And for the cluster itself, Phoenix ships the piece that finds the other nodes:

children = [
  MyApp.Repo,
  {DNSCluster, query: System.get_env("DNS_CLUSTER_QUERY") || :ignore},
  {Phoenix.PubSub, name: MyApp.PubSub},
  MyApp.Meter.Supervisor,
  MyAppWeb.Endpoint
]

On Fly.io, DNS_CLUSTER_QUERY is your app’s internal name. Without connected nodes, PubSub still works locally and every node quietly believes it is alone, which looks exactly like undercounting. Assert on it: log Node.list() at boot and alert when its length is not what you deployed.

Testing the cluster case without a cluster

Most of the model is testable in one VM, because “two nodes” is really “two ETS tables and two flushers against one database”:

test "two flushers sum instead of overwriting" do
  a = start_store!(:node_a)
  b = start_store!(:node_b)

  for _ <- 1..400, do: Store.bump(a, org.id, :api_requests)
  for _ <- 1..622, do: Store.bump(b, org.id, :api_requests)

  Flusher.flush(a)
  Flusher.flush(b)
  Flusher.flush(a)

  assert Usage.total(org.id, :api_requests) == 1_022
end

The third flush is deliberate. A flush with nothing new must be a no-op, and this is the test that catches an absolute write pretending to be a delta. For the retry path, call flush_batch/2 twice with the same batch id and assert the total does not move. Then, once, run it for real with :peer.start_link/1 and two connected nodes, because gossip and node-down handling are the parts that only fail when the network is genuinely involved.

Or use the one dependency

Everything above is a few hundred lines, plus the concurrency tests, plus the shutdown flush you will forget the first time. It is the free MIT core of Aurora Meter, which ships the delta model, the gossip, the idempotent flush batches and the cold seed:

# mix.exs
{:aurora_meter, "~> 1.0"}

AuroraMeter.track(org, :api_requests)

<.usage_meter tenant={@org} feature={:api_requests} />

The core is on GitHub under MIT, with no email gate and no expiry, and it keeps your usage data in your own Postgres. Aurora Meter Pro is the layer for when Stripe should start charging: usage reported to Stripe Billing Meters, billing periods aligned to the subscription rather than the calendar month, historical rollups and CSV export, quota alerts and reconciliation. It is licensed per app from $99 a month, with a 14-day free trial claimed once per customer, and the details are on the pricing page. If you are also standing up the app around it, phx_saas gives you the auth, billing and admin scaffolding to hang the meter on.

Summary

Cluster usage metering comes down to four decisions. Count locally in ETS so the hot path never blocks. Flush deltas rather than absolutes, so nodes add up in Postgres instead of overwriting each other. Make each flush an idempotent batch, so a retry after an uncertain commit cannot bill twice. Gossip unflushed deltas over the PubSub you already run, so a read on any node is the true total minus a known, bounded amount.

What you get is a number the whole fleet agrees on, at ETS speed, with a staleness you can state in a sentence. What you give up is exactness at the instant a cap is hit, which is the right trade for anything you price as usage, and the wrong one for anything you can count with a unique index instead. For the money side of the same model, the usage-based billing guide picks up where this leaves off.