This has been a busy week for one open-source chat app. DHH had AI agents port 37signals’ Campfire from Rails to Rust, Go and Elixir and shared benchmarks with Rust on top. The Elixir community dug in and found a large gap in effort between the ports. Pull requests followed.
Now there’s a new kind of test, and it asks a different question: not how fast, but how reliable.
The headline benchmark has a blind spot
The original benchmark measured throughput: requests per second at a fixed number of clients. It’s a useful number, but as Zach Daniel explained in his reliability pull request, it “measures throughput at 16 clients and aborts on the first failure, so it can’t show how an implementation behaves past that point.”
For a chat app, behavior past that point is the whole question. What happens when lots of people post at once? Do messages still arrive?
What the reliability test does
The PR adds a bin/reliability script to the shared verification repo. It uses the same production images, data, CPU limits and response checks as the main benchmark, but records failures instead of stopping. It runs two profiles:
- Overload: the main read routes at 64 and 256 clients, reporting valid requests per second, errors and latency percentiles.
- Cable: 1,000 subscribers to a busy chat room, counting how many subscribed, how many the server disconnected, and how many posted messages reached every subscriber. It runs once with 4 clients posting and again with 64.
Zach ran it on a 16-vCPU machine with the server limited to 4 CPUs. He notes that machine is slower per core than the one used for the published report, so the numbers should only be compared with each other.
The results
Chat delivery, 1,000 subscribers, two rounds
| Posters | Disconnected | Delivered to all | Messages/s | |
|---|---|---|---|---|
| Rails | 4 | 0, 0 | 128/128, 139/139 | 8.5 |
| Rails | 64 | 0, 0 | 191/191, 192/192 | 8.9 |
| Elixir | 4 | 0, 0 | 476/476, 492/492 | 32.2 |
| Elixir | 64 | 0, 0 | 576/576, 602/602 | 35.2 |
| Rust | 4 | 0, 0 | 3,874/3,874, 3,744/3,744 | 253.7 |
| Rust | 64 | 756, 404 | 66/23,192, 97/11,156 | 5.4 |
At light load, Rust is in a different league: about 254 messages per second delivered, against 32 for Elixir and 8.5 for Rails.
At 64 posters, it flips. Rust disconnected 756 subscribers in one round and 404 in the other, and delivered only 66 of 23,192 messages to every client in the first round. The PR explains why: Rust “disconnects subscribers that fall 256 messages behind,” as documented in its cable server. Rails and Elixir kept every subscriber and delivered every message, with higher latency.
José Valim, Elixir’s creator, summed it up: “Rust is dropping more than 99% of messages under load, delivering fewer messages than Rails or Elixir.” (post)
Read routes under overload
| Errors / invalid | Valid req/s at 256 clients | p99 at 64 → 256 clients | |
|---|---|---|---|
| Rails | 0 / 0 | 1,791 | 171ms → 1,108ms |
| Elixir | 0 / 0 | 1,943 | 48ms → 154ms |
| Rust | 0 / 0 | 18,229 | 7ms → 26ms |
On plain page loads, Rust is far ahead even under overload, and all three stayed correct. This isn’t a story where Rust is bad. It’s a story where one design choice, dropping slow subscribers, trades delivery for speed.
A design choice, not a language verdict
This is the most important point. Disconnecting subscribers that fall too far behind is a reasonable choice in some systems. It protects the server from slow clients. It’s a bad fit for a chat app where people expect every message to arrive.
The Rust code could be changed to buffer more or reconnect. The Elixir code could be tuned to go faster. Neither result is fixed by the language. What the test shows is that a throughput number alone can hide a reliability problem, and that you only find it by measuring failure on purpose.
Where things stand
The work is still open. José has opened several PRs on the Elixir port, including one to remove Redis and run jobs and Action Cable delivery in process. He’s asked that Elixir be allowed to “deviate from the Rails version in the same way Express/Go/Rust/ do” (post), and that request is the crux of a fair comparison.
It’s also worth keeping the bigger picture. Brian Cardarella, a long-time Elixir advocate, said: “I don’t believe that DHH did the benchmark to dunk on anybody” (post), and that his only issue is that “everybody is hung up on the numbers and not that DHH created all of those implementations without knowing those stacks. That is the value.” (post)
Meanwhile, Phoenix keeps moving. José announced an upcoming LiveView feature where any LiveView or LiveComponent can subscribe to a PubSub topic and have LiveView manage the subscription for you (PR).
What to test before you trust a benchmark
If you’re choosing a stack, a vendor or an architecture, especially for real-time work:
- Push past the happy path. Run at several times your expected load and record failures instead of aborting.
- Measure delivery, not just acceptance. A server that accepts a message hasn’t delivered it.
- Watch the tail. p99 latency under load tells you more than averages.
- Look for documented limits. Buffer sizes, timeouts and disconnect rules are design decisions with real costs.
- Test reconnects. Clients drop. What happens next matters.
Fast is great. Fast and reliable is what users actually need. If you want help testing a system before you bet on it, book a call.
Rust logo © Rust Foundation, used under CC BY 4.0.
