Record the facts you invoice

A buffered counter is the right answer for most things you measure and the wrong answer for the handful you invoice one by one. This guide is about that handful: how to record a fact so that it survives a crash, so that a retry does not bill twice, and so that changing your mind later leaves a trail instead of a smaller number.

It assumes you have read Count what customers use. Everything here is in the free core.

#Two ways to count, and why you pick one per feature

AuroraMeter.track/4 adds to an in-memory counter. It is fast because nothing is written while your request is running, and the cost of that is what sits in memory when a machine dies.

AuroraMeter.record/4 writes a row inside a transaction before it returns. It is slower, it needs your database to be up, and what it gives you is an identity per fact: something you can look up, retry safely, and correct.

You do not choose between them globally. You say, per feature, where that feature’s commercial quantity comes from:

config :aurora_meter, feature_sources: %{tokens: :events, api_requests: :buffered}

Anything you do not list is :buffered, so this config says one thing: tokens are recorded, everything else is counted.

One feature, one source. This is the rule the config exists to enforce. If tokens were both counted in ETS and recorded as events, the flusher would write a counter row and the exporter would stage the events, and the customer would be billed for both. Aurora Meter refuses the ambiguity rather than picking a side.

It refuses it twice over. Calling AuroraMeter.track/4 for a feature whose source is :events raises before it touches ETS, naming the feature and telling you which config line would change its mind. And naming one feature in both the old :durable_features list and feature_sources fails the boot.

#Writing a fact

AuroraMeter.record(org, :tokens, 1_840,
  id: "gen:" <> request_id,
  occurred_at: DateTime.utc_now()
)
# {:ok, %AuroraMeter.Event{}, :inserted}

:id and :occurred_at are both required and neither has a sensible default. The id is yours: one to 128 bytes, unique for this tenant across every feature. occurred_at must be a DateTime in Etc/UTC, and it may not run further ahead of the node clock than events_future_tolerance allows.

Two optional maps ride along. :dimensions is for things you slice reports by (at most 32 string keys, scalar values), and :metadata is for everything else (up to 16 KiB of JSON). Both are string keyed.

#Choosing the id

This is the decision the whole design rests on, so spend a minute on it.

The id has to be something your own code can produce again from the same inputs. A request id, a job id, a row’s primary key. Never a timestamp and never a fresh UUID generated at the call site, because the point of an identity is that the retry carries the same one.

A useful habit is to make one string do three jobs: the primary key of your own row, the credit hold’s reference, and the event id. Then a customer support question about one of them answers the other two.

#What a second call with the same id does

AuroraMeter.record(org, :tokens, 1_840, id: "gen:abc", occurred_at: at)
# {:ok, event, :inserted}

AuroraMeter.record(org, :tokens, 1_840, id: "gen:abc", occurred_at: at)
# {:ok, event, :duplicate}

The second call writes nothing and tells you so. That is what makes the call safe to retry after a timeout: you do not have to know whether the first attempt landed.

Change anything about the payload and you get a different answer:

AuroraMeter.record(org, :tokens, 2_000, id: "gen:abc", occurred_at: at)
# {:error, {:conflict, %AuroraMeter.Event{quantity: 1_840}}}

Nothing is written, and the error carries the event that is already stored, so you can see what it disagrees with. Two different facts under one identity is a bug in the caller, and the library will not quietly pick one.

#The error shapes worth handling

case AuroraMeter.record(org, :tokens, qty, id: id, occurred_at: at) do
  {:ok, _event, :inserted} -> :ok
  {:ok, _event, :duplicate} -> :ok
  {:error, {:conflict, stored}} -> Notewell.Ops.report_conflict(id, stored)
  {:error, {:invalid, problems}} -> {:error, problems}
  {:error, {:unavailable, reason}} -> {:error, reason}
end

{:invalid, problems} is a list of {field, reason} pairs, so a bad id or a non UTC timestamp tells you which one and why. {:unavailable, reason} covers a database that timed out and the case where too many callers hold an open record transaction at once, which answers {:unavailable, :overloaded}. There is no fallback to buffered tracking on either: a call that could not record a fact does not quietly record a different kind of thing instead.

#Recording inside your own transaction

If you are already in a transaction of your own, the event commits with it. It comes back marked conditional, and you tell Aurora Meter when your transaction landed:

{:ok, %{event: event}} =
  Repo.transaction(fn ->
    {:ok, row} = Notewell.Generations.insert(attrs)
    {:ok, event, _} = AuroraMeter.record(org, :tokens, attrs.tokens, id: row.id, occurred_at: at)
    %{row: row, event: event}
  end)

AuroraMeter.Events.after_commit(event)

after_commit/1 is what releases the downstream work (projections, export intents) that must not happen if your transaction rolled back. It is a no op for an event recorded outside a transaction, so calling it always is safe.

#Batches

AuroraMeter.record_batch([
  %{tenant: org, feature: :tokens, quantity: 900, id: "gen:a", occurred_at: at},
  %{tenant: org, feature: :tokens, quantity: 120, id: "gen:b", occurred_at: at}
])
# {:ok, [{event_a, :inserted}, {event_b, :inserted}]}

Up to 500 elements. The whole batch is one transaction, so one conflicting element rolls all of it back and tells you the index: {:error, {:conflict, 1, stored}}. Repeating an id inside one batch with two different payloads is refused before any database call.

#Corrections

You billed for 1,840 tokens and 400 of them were a retry you should not have charged for. You do not want a second record, and you certainly do not want a negative track, which would move a counter that is not even the source for this feature.

AuroraMeter.correct(org, "gen:abc", 400, id: "fix:abc:1")
# {:ok, %AuroraMeter.Event{}, :inserted}

Read the arguments carefully, because two of them surprise people.

The third argument is the size of the reduction, as a positive number. Not the new total, and never a negative. You are saying “take 400 off”, not “make it 1,440”.

The :id is the correction’s own identity, not the original’s. Corrections are facts too, and they are retried the same way, so they need their own id.

Four rules follow from corrections being append only:

• The original is never modified. Both rows stay, and a report can show you what was claimed and what was taken back.
• The corrections of one original can never take back more than the original claimed. Try, and you get {:error, {:invalid, [quantity: :exceeds_original]}}. The check runs with the original row locked, so two concurrent corrections cannot both slip under the line.
• A correction belongs to the original’s period, inherits its feature and its plan attribution, and never reprices anything. A correction made in October against a September fact lands in September.
• You cannot correct a correction. That is {:error, {:invalid, [original: :is_correction]}}. Correct the original again instead.

The duplicate check runs before the bound check, deliberately. Retrying a correction you already made answers :duplicate rather than pretending you are now over the line.

:dimensions and :occurred_at are refused here, not ignored. A correction that could move the timestamp could move the fact into another period, and then the September invoice depends on when somebody noticed the mistake.

#Replacing a fact outright

When the quantity was not merely too high but wrong in several ways, replace/4 writes the correction and the new fact together:

AuroraMeter.replace(org, "gen:abc", %{quantity: 1_440, occurred_at: at}, id: "fix:abc:2")
# {:ok, %{correction: correction, replacement: replacement}, :inserted}

The correction takes back whatever is left of the original, and the replacement is a new event whose id defaults to your correction id with ~r appended. A feature key in attrs must match the original’s; you are replacing a fact, not moving it to another product.

#Reading events back

AuroraMeter.Events.get(org, "gen:abc")
# {:ok, %AuroraMeter.Event{}} | {:error, :not_found}

AuroraMeter.Events.total(org, :tokens, period_start)
# 1_440

AuroraMeter.Events.count(org, :tokens, period_start)
# %{quantity: 1_440, events: 2}

For a report or an export, stream in sequence order rather than paging by id:

AuroraMeter.Events.stream(tenant: org, feature: :tokens, after_seq: cursor, limit: 500)

:from and :to filter on occurred_at and are half open, like every other interval in the library.

#What this costs you

Be honest with yourself about the trade before you move a feature to :events.

Every recorded fact is a transaction on your database on your request path. The library bounds the damage (a timeout returns rather than hanging, a concurrency ceiling returns :overloaded rather than queueing your whole application behind Postgres) but it cannot make the write free. Record the things you invoice one by one. Count the rest.

#What to read next

• Ship it: the production checklist covers what to watch once this is running.
• Write your plans down, then enforce them is the other half: recording what happened, and deciding what is allowed.