A 1ms partition is enough to corrupt a node
· 5 min read
- elixir
- distributed-systems
- pubsub
Messages are guaranteed to be delivered as long as the distribution link between the nodes is active, but it gets tricky when a partition occurs - that means both nodes online, but the link between them is down.
Phoenix.PubSub.PG2 delivers a broadcast to the nodes connected at that instant. No buffer, no ack. If a node is unreachable when you broadcast, the message is gone for that node - and nobody finds out.
Don’t get me wrong - for a typing indicator or a presence blip, losing one costs nothing and the next update fixes it anyway. But when a missed message leaves a node permanently wrong - a cache invalidation, an event feeding derived state - fire-and-forget is not something you can build on.
Here is what a 1ms partition does:
Fire and forget
Node A - no buffer, no acks
Node B
B comes back and carries on as if nothing happened. No error, no gap detection - msg 2 and msg 3 simply never existed as far as B is concerned. If those messages were cache invalidations, B now serves stale data forever. If they fed a projection, B’s derived state is quietly wrong.
That’s the part that bothers me: not that messages can be lost - networks lose things - but that the loss is silent. You can’t retry what you don’t know you missed.
Adding an at-least-once guarantee on top of PubSub
Picture the broadcasting node with a small notebook.
- The notebook has a fixed number of pages (
buffer_size, say 5), and one message goes on one page. Fill the last page and you loop back to the first, writing over whatever was there. So the notebook always holds the 5 most recent messages and nothing older - that’s all “page” means here. - Every message also gets a number that never resets: msg 1, msg 2, msg 3, and on forever. That counter is
write_cursor. The pages cycle; the numbers don’t. - Next to each other node, A keeps a bookmark (
read_cursors[node]): “this node has confirmed everything up to msg N.” It moves only when that node writes back “got it”. No reply, no move - so anything unconfirmed is simply sent again next time. That is the entire at-least-once guarantee.
Notebook (5 pages) - press the buttons
write_cursor:
So: message numbers count up forever, pages get reused. Subtract a node’s bookmark from the current message number and you get one number - how far behind it is. That number answers both questions at once: what does the node still need, and is it still in the notebook?
Three cases, and that’s the whole decision:
bookmark = read_cursors[node] # how far this node has confirmed
lag = write_cursor - bookmark # how far behind it is
lag == 0 -> :caught_up # it has everything, send nothing
lag <= buffer_size -> {:messages, …} # still in the notebook, resend those pages
lag > buffer_size -> {:expired, …} # written over, can't be resent
The same partition, this time with a notebook
Step through the partition from the top of the post and watch the bookmark. The second tab walks the other case, where B stays away too long:
Node A - notebook (5 pages)
Node B
bookmark:
No gap: B gets 2 and 3, in order, before it ever sees 4.
And when B stays away so long that the messages it needed are written over, A does not quietly carry on and let B think it is up to date. B is told “I lost your place, reload from the database” - it knows its state is stale and fixes it. That’s the actual guarantee:
Either you receive every message in order, or you are told you fell behind.
Getting message 3 means you already have 1 and 2. Never a silent hole.
The cost is duplicates. If a batch is delivered and handled but the ack is lost, the cursor doesn’t advance and the batch is re-sent. Handlers have to tolerate that - send absolute state rather than deltas, or dedupe by message id.
Demo in action
ScoreBoard is three nodes on one live match. Take a node offline and its column freezes while the peers buffer for it (the orange box counts what’s held); bring it back and the backlog replays in order. Overflow the ring buffer - 5 here - and the box turns red instead: that peer gets {:cursor_expired, node} and reloads from the source of truth. Converged either way.
Performance: one ack per batch, not per message
Acking every message separately would mean a network round-trip each; instead a flush is one acked call per node carrying everything written since the last one. batch_interval is how long a producer collects writes before flushing them. batch_interval=0 means flush as soon as possible - but that is still not one call per message: a producer waits for the previous call to a node to be acked before sending the next, and everything written during that wait goes out together. So the faster you publish, the bigger the batches get.
On a Fly.io cluster (fra, performance-4x, v0.2.3, pool_size=1), 3 nodes, 30 B payload, one publisher, every node receiving every message:
| batch_interval | msg/s | remote p50 | calls/s per node | msgs/call |
|---|---|---|---|---|
| 0 | 100 | 1.6 ms | 100 | 1 |
| 0 | 65,911 | 1.3 ms | 953 | 69 |
| 100 ms | 100 | 52.8 ms | 9 | 11 |
| 100 ms | 99,820 | 94.7 ms | 9 | 10,697 |
Low latency and low CPU are the two things you cannot have at once. batch_interval=0 gets messages out 25 to 70 times faster, but at the cost of far more calls and higher CPU usage; a 100 ms timer sends a handful of big calls for a fraction of the CPU, and reaches a higher ceiling, but every message waits. Payload size is the other lever - 200 B costs 20-55% against 10-30 B, so send small deltas, not whole objects.
Full numbers: throughput, steady rate.
When not to use it
The buffer is in memory and bounded. It closes the brief-partition gap; it is not a durable log. Want persistence across a full cluster restart, replay from disk, cross-language consumers, or huge backlogs? Run a real broker. Want reliable cross-node messaging on a BEAM cluster you already operate, without standing up a message broker to get it? That’s the gap this fills.
It’s also not a drop-in replacement - at-least-once costs more than fire-and-forget. Keep the default PubSub for ordinary broadcasts and run EchoPubSub alongside it, for the streams where a missing message quietly corrupts state.
Inspired by the work of Eric Newbury on phoenix_pubsub_buffered.
Cheers!