The Part of Elixir I Kept
I built the textbook answer to agent orchestration — OTP supervision trees — and it worked. Then I deleted it, and kept what mattered.
In March I built the textbook answer to my problem, and it worked. This month I stripped the last references to it from the design docs. This is the story of the four months in between, and of the parts that survived.
Part one covered the methodology: constraints compose, opinions don’t. This is the architecture story underneath it — where the loop that enforces those constraints should live, and how I got that wrong before getting it right.
The problem OTP was born for
rosary started as a god-file reconciliation loop — everything in memory, all decisions in one place — and it kept breaking in three specific ways. Crashes lost state. Stuck agents sat undetected until morning. Continuations vanished between pipeline phases.
If you list those failures for anyone who has touched the BEAM, they’ll interrupt you before you finish: crash recovery, process monitoring, and durable handoff are precisely what Erlang/OTP has solved for thirty years. Supervision trees restart what dies. Monitors notice what hangs. The runtime was built by people whose telephone switches were not allowed to stop working. When OpenAI’s Symphony shipped in Elixir, it read like confirmation — the industry’s newest agent orchestrator was standing on the industry’s oldest reliability runtime.
So I built the conductor: an OTP application where every agent ran under a supervisor, dispatched over ACP ports instead of shelled-out claude -p calls. Completion detection was instant — a process exit is a message, not something you poll for. Retries backed off exponentially and deadlettered after max attempts. The design doc described it as the persistent version of Claude Code’s agent teams, and that’s what it was.
It worked. That’s what makes this story worth telling — nothing that follows is about Elixir failing.
The supervisor had a supervisor
The trouble was never the runtime. It was what the runtime cost to ship.
rosary is a local-first tool: it runs on my machine, against my repos, overnight. The conductor turned that into a deployment: a BEAM release in a container on Fly, running beside the rosary server and a tunnel daemon, with an entrypoint script whose job was to kill everything if any process died so the platform could restart the machine.
Read that again slowly. The supervision runtime was being supervised by a shell script, which was being supervised by a cloud platform, so that agents on my laptop could be supervised by the BEAM. Every layer was individually reasonable. The stack of them was absurd — and at the time, the thing at the bottom of all that machinery was a single local process.
Elixir turned out to be an excellent orchestrator of orchestrators. That’s a real niche, and if rosary were a hosted, multi-tenant service, the conductor would probably still be alive. But requiring a daemon, a container, and a platform to babysit what is locally one process meant the operational surface had outgrown the thing it protected.
What survived the deletion
Here’s what I noticed when I pulled the conductor out: nothing about its design had to leave. Only the runtime did.
Supervise and restart with backoff. Deadletter after N failures instead of retrying forever. Detect completion by event, not by polling. Hold desired state somewhere durable and run a single loop that converges reality toward it. None of those are Elixir features — they’re invariants, and OTP is one vehicle for them. Kubernetes is another: nobody runs the BEAM in a cluster control plane, yet every controller is a reconcile loop doing exactly what a supervision tree does, converging on declared state and restarting what drifts.
So the invariants moved into rosary’s Rust reconciler, where the state lives in the bead database and the loop scans, dispatches, verifies, and syncs. The retry budgets and deadletters came along. So did event-driven completion. What didn’t come along was the second runtime, the container, and the babysitting script.
Four months later that reconciler has grown into most of rosary — on the order of seventy thousand lines of Rust — and it has closed 1,600 beads since March, almost all of them overnight while I slept. The conductor’s ideas are carrying more load now than they ever did in the conductor.
Pick the invariant, not the runtime
The mistake I’d made has a general shape. I had a set of invariants I needed — supervised convergence toward declared state — and I reached for the runtime most famous for them, without asking whether the runtime’s deployment shape fit the product. Runtimes are vehicles. When the vehicle costs more to operate than the cargo is worth, the cargo doesn’t have to go down with it.
The part of Elixir I kept is the part that was never Elixir-specific: the conviction that failure is normal, that recovery must be structural rather than heroic, and that one loop converging on declared state beats a thousand clever handlers. That conviction now runs in Rust, unsupervised, every night.
rosary is open source: github.com/agentic-research/rosary · rosary.bot. A Gleam re-telling of the conductor — typed OTP — may happen someday; some vehicles deserve a second trip.