In part 1 we laid out DHH’s argument: AI agents ported 37signals’ Campfire chat app from Rails to Rust, Go and Elixir, Rust came out fastest, and if agents write the code, languages they can’t write well out of the box will struggle. Then we looked at the effort gap between the ports and what a reliability test showed.
Since then, more ports have arrived (Express, Laravel, Django and C), and DHH has moved the benchmarking into a shared verification repo. This morning, José Valim, Elixir’s creator, posted what he found when he asked why PRs speeding up Elixir were being turned down while similar ones were accepted elsewhere. We went through the repos ourselves to check. The short version: he’s right, and the evidence is in the repos.
What José said
From his post this morning:
It turns out that, not only each project had a different starting point and a different prompt, but also each project also has a different set of verifications that they are required to pass!
For example, Elixir is the only guided to ‘preserve the pinned Rails reference behaviour’, the only with a explicit verification that we are using Redis and draining its queue (so we cannot remove it while others can), and the only one actively enforcing it.
He linked a write-up his agent produced. We didn’t take it on faith. Everything below we checked directly in the repos.
What the repos actually say
Each port has an AGENTS.md file: the standing instructions the coding agents follow.
Rust (AGENTS.md): parity with Rails “is done,” and “The app may now diverge from Rails where that makes it faster or better.” Its README says “One campfire executable replaces Ruby, Puma, Redis, Resque and Thruster,” and that Redis and Resque “are replaced by in-process queues.” It’s upfront about the cost: “Queued pushes and webhooks are lost on a crash.”
Go (AGENTS.md): a “Port of the Rust Campfire,” told to preserve “the behavior of the Rust app,” not Rails. Its README: “As in Rust, background queues are bounded and in-process,” and “a process crash can lose queued jobs.”
Express (AGENTS.md): reads the Rails reference for behavior and preserves the schema, storage and cookie signing. Jobs run on “A separate leased SQLite queue,” per its README.
Elixir (AGENTS.md): “Preserve the pinned Rails reference behavior, SQLite schema, storage layout, signed/encrypted cookies, frontend, and Action Cable protocol.”
And the Elixir benchmark runner enforces it. In bench/run, after each sample, the Elixir app must report a Redis/Resque backend with a drained queue, or the run fails:
if app in {"reference", "baseline", "elixir", "candidate"}:
assert jobs.get("backend") == "redis_resque" and jobs.get("drained") is True, jobsThe same runner also refuses to benchmark at all unless Elixir’s parity ledger is complete, every contract is marked verified, and the last full verification ran on the exact current source (bench/validate.py). We didn’t find an equivalent pre-benchmark gate in the other ports.
The comparison
| Rust | Go | Express | Elixir | |
|---|---|---|---|---|
| Behavior baseline | Rails, but “may now diverge” | The Rust port | Rails reference | ”Preserve the pinned Rails reference behavior” |
| Redis / Resque | Replaced, in-process | Replaced, in-process | Replaced, leased SQLite | Required; drain checked by runner |
| Thruster | Replaced | Not stated | Replaced by a Thruster-style in-memory cache | Kept in the original port |
| Jobs lost on crash? | Yes, documented | Yes, documented | No (durable queue) | No (Redis) |
| Pre-benchmark parity gate | None found | None found | None found | Required |
| PRs merged / total | 42 / 47 | 2 / 11 | 1 / 4 | 1 / 13 |
| Commits on main | ~437 | ~40 | ~43 | ~45 |
PR and commit counts are from the GitHub API as of this morning. The Rust repo is also where DHH did most of his own tuning, so its PR count mostly reflects his work, not outside contributions.
DHH’s stated reasons
To be fair, the closed Elixir PRs weren’t waved away. The reviews posted on them are specific and technical.
On Zach Daniel’s PR, the review said the rewrite “removes the durable Redis/Resque job path and live Rails interoperability that existing installs depend on,” and that a narrower PR, #4, was merged instead. On PR #2, PR #5 and PR #7, the reviews found real cache-invalidation problems, and on #2 and #7 the authors agreed. On PR #7, the author wrote: “Agreed on all three points.” Several reviews also note that the claimed speedups “were not independently reproduced.”
DHH’s public line has been consistent: “Happy to take any improvements that actually hold up to verification!” (post). And the Elixir port did get faster. The latest performance review shows its room page at 5,350 requests per second, ahead of Rails at 4,101.
Durable jobs and compatibility with existing installs are good reasons to turn down a change. We’d make the same call on a client project.
Where the rules don’t line up
The problem isn’t any single review. It’s that the same reasons weren’t applied to every port.
- Durable jobs. Zach’s PR was turned down partly for removing “the durable Redis/Resque job path.” Rust and Go have no durable job path, and their own READMEs say queued jobs are lost on a crash. That was accepted for them.
- Rails interoperability. Rust’s instructions say it “may now diverge from Rails where that makes it faster or better.” Elixir’s say “Preserve the pinned Rails reference behavior.”
- Who replied to Zach. On his closed PR, Zach answered: “I modeled it after the rust implementation which also does not do those things by my understanding.” Based on Rust’s own README, his understanding was correct.
- The gate. Only Elixir has a runner that requires Redis and blocks benchmarking until a full parity run passes on the exact source.
None of this requires bad intent to explain. Each port was set up at a different time, with a different prompt and different instructions to its agent, and those instructions then judged incoming PRs. José makes the same point: “each project had a different starting point and a different prompt.” The result is still a table where Elixir runs a heavier architecture under stricter rules, sitting next to ports that were allowed to drop that architecture.
The headline numbers make the cost visible: in the latest table, Express serves the room page at 42,636 requests per second and Elixir at 5,350. Some of that gap may be Elixir. But Express is allowed an in-process cache and no Redis, and Elixir’s runner requires Redis. Until the rules match, we can’t tell how much.
What a fair benchmark needs
We said this in the effort post, and it applies even more now:
- One rulebook. Either every port must preserve Rails’ Redis/Resque architecture, or every port may replace it. Not some of each.
- One set of checks. If a parity gate or a job-drain check is required for one port, require it for all of them, through the shared runner.
- Published instructions. The
AGENTS.mdfor each port is part of the benchmark and belongs next to the results. - Same-session reruns. The current review notes that Elixir, Go, Rust and C “retain exactly their independently verified figures from the previous published comparison” and weren’t rerun. Rerun everything together.
DHH’s underlying question, which languages agents write well out of the box, is still a good one. Answering it takes the same rules for every language. If you’re running your own comparison of stacks or vendors, we can help set it up fairly.
