Two days ago we wrote about DHH’s Campfire shootout: AI agents ported 37signals’ Campfire from Rails to Rust, Go and Elixir, and Rust came out fastest. His argument was that if agents write the code and nobody reads every line, the fastest language agents can write well wins.
Since then, people have dug into the repos. What they found is a good lesson in how to benchmark anything, and especially code written by agents.
What the commit history shows
The three ports are public on GitHub. We counted commits on each one up to the moment DHH posted his comparison on October 4:
| Port | Commits before the benchmark post | Authors |
|---|---|---|
| Rust | 400 | 3 |
| Go | 2 | 1 |
| Elixir | 2 | 1 |
The Rust repo was created on September 26, so those 400 commits came over about a week. The Elixir port’s first commit is titled “Port Campfire to Elixir with Rails parity and Rust benchmarks,” and it landed the morning of the same day the results were posted.
Others found the same gap. Developer @jskalc summarized it: “Rust’s first port from Ruby: 4.4× Rails. Elixir’s: 3.3×. Then Rust got 398 commits and 18 perf PRs. Go was ported from that tuned Rust. Elixir got one small tuning pass.” (post)
José Valim, Elixir’s creator, looked at the Rust repo’s own early numbers and found that they “are in same order of magnitude as Elixir/Go (sometimes worse)” (post).
So the first-pass ports were close. The big gap came later, after a week of tuning that only one port got.
What happened when Elixir got the same treatment
DHH invited patches, and people sent them. Two pull requests are worth reading.
Zach Daniel’s PR went after the design. Among other changes, it “Replaced the single database process with a pool of read-only connections plus one writer,” moved message caching from Redis into ETS, and fixed a bug where Bandit closed WebSockets that hadn’t sent anything for 60 seconds. Action Cable clients mostly listen, so idle chat tabs kept getting disconnected.
A second PR from GitHub user lau built on that and posted these numbers against the tuned Rust port, in requests per second:
| Route | Elixir (improved) | Rust (improved) |
|---|---|---|
| Room page | 17,237 | 26,796 |
| Messages page | 25,238 | 28,335 |
| Sidebar | 32,121 | 27,538 |
| Search | 26,790 | 25,488 |
| Post a message | 1,995 | 6,101 |
Rust still wins on writes and on the room page. But on two of the five routes, Elixir is ahead. That’s very different from the original chart.
@jskalc also noted a limit of the benchmark itself: it has no writes on the read routes, and he believes both tuned versions are hitting cache nearly every time, “so it’s not very realistic” (post).
In fairness to DHH
It’s worth being fair here. DHH has said he has no stake in the outcome: “I don’t have a horse in this race. If you can get the elixir to go faster, awesome! Send a patch and I’ll merge it” (post). He’s also said he didn’t spend time on the other ports because he isn’t interested in those stacks: “I kicked off six conversions of an open source app to see where the agents would take them” (post).
And Brian Cardarella, who runs the Elixir consultancy DockYard, pointed out that the benchmark “showed how much faster Elixir is than Ruby” (post).
The point isn’t that anyone cheated. It’s that the result reflects effort as much as language.
How to run a fair benchmark
Whether you’re choosing a language, a framework or a vendor, the same rules apply.
Give every option the same budget
Same prompt, same number of tuning rounds, same hours. If one option gets a week and another gets an afternoon, you’re measuring the week.
Give every option the same freedom
The Elixir port was told to keep Rails parity, including Redis and the Rails-style design. The Rust port was free to diverge. Either hold everyone to the original design or let everyone redesign.
Compare first passes and tuned versions separately
The first-pass comparison answers “how good is the agent out of the box?” The tuned comparison answers “how fast can this get?” Both are useful. Mixing them isn’t.
Measure what users feel
Throughput on cached reads is easy to measure. For a chat app, you also care about writes, latency under load, and whether messages actually arrive.
Publish the method with the numbers
Commit counts, prompts and tuning steps should sit next to the chart. Otherwise readers can’t tell what’s being compared.
What this means for agent-led rewrites
DHH’s premise still deserves attention: agents make rewrites cheap, and how well agents write a language out of the box matters. But “out of the box” has to mean the same thing for every option.
If you’re planning to have agents rewrite part of your stack and want to compare options fairly before committing, that’s work we do. Book a call.
Rust logo © Rust Foundation, used under CC BY 4.0.
