What Retrying Costs — Not the Side Calling, the Side Receiving

At the end of Orders with the Till Gone we wrote this.

There is no automatic retry. The harness reconnected explicitly. A real app would try periodically in the background, and that period and backoff were not decided here.

This piece decides that period. And while deciding it, measures what happens when you decide it badly.

This is not a specific business. We constructed the situation of a payment gateway going down briefly. The code, screens and logs actually ran.

Counting from the receiving side

Most retry writing is the caller's story — how do I get my request through.

This sample's server stands on the other side. It counts how many attempts arrived, and how many of those landed inside the same 200 milliseconds.

/// That second number is the one that matters. A gateway that is briefly down
/// does not care that you tried again. It cares how many of you tried again at
/// the same instant, because that is what keeps it down.

A briefly-dead gateway is not interested in the fact that you tried again. What it cares about is how many of you tried again at the same instant. That is what keeps it dead.

What gets written when you are in a hurry

/// Try again immediately. This is what gets written when the retry is added in
/// a hurry, and it is the strategy that turns a short outage into a long one.
class NoBackoff extends Backoff {
  @override
  int waitBefore(int n) => n == 1 ? 0 : 20;
}

RETRY IMMEDIATELY: 4 callers, 64 attempts total, 4 accepted, peak 18
  caller 1 attempt 1: straight away -> DOWN at 12ms
  caller 1 attempt 2: after 20ms -> DOWN at 58ms
  ...
  caller 1 attempt 16: after 20ms -> ACCEPTED at 708ms

It got through. It survived a 700-millisecond outage and all four were charged.

And the gateway was hit 64 times in the meantime.

Double each time, then shake it

@override
int waitBefore(int n) {
  if (n == 1) return 0;
  final base = 60 * (1 << (n - 2)); // 60, 120, 240, 480 ...
  // Full jitter: anywhere in [0, base]. Spreading matters more than being
  // punctual — nobody is waiting on an exact millisecond here.
  return _rng.nextInt(base + 1);
}

EXPONENTIAL + JITTER: 4 callers, 25 attempts total, 4 accepted, peak 13
  caller 1 attempt 1: straight away -> DOWN at 13ms
  caller 1 attempt 2: after 53ms  -> DOWN at 70ms
  caller 1 attempt 3: after 87ms  -> DOWN at 159ms
  caller 1 attempt 4: after 143ms -> DOWN at 305ms
  caller 1 attempt 5: after 389ms -> DOWN at 697ms
  caller 1 attempt 6: after 449ms -> ACCEPTED at 1150ms

Same outage, same people, same result. All four were charged.

Attempts went from 64 to 25. Times 2.6 fewer.

Doubling and shaking are different jobs

This is what this piece wants to say.

Everybody knows exponential backoff — 60, 120, 240, 480. Jitter often gets left out. But with one caller jitter makes no difference, and with several, jitter is everything.

The reason is simple. If four failed at the same instant, then without jitter four come back at the same instant. Four at 60ms, four at 180ms, four at 420ms. The gateway keeps taking the same spike that just knocked it over.

Look at the log above: the waits are 53, 87, 143, 389, 449 — none of them 60, 120, 240 or 480. They are shaken values.

The verification confirms that.

# Jitter must actually produce different waits, otherwise the four callers
# come back together and the whole point is lost.
WAITS=$(grep -o "after [0-9]*ms ->" captures/run.log | sort -u | wc -l | tr -d ' ')
[ "$WAITS" -ge 2 ] || { echo "   every wait was identical — jitter is not doing anything"; exit 1; }