Many Agents, One Repository: Coordination Without a Supervisor

The first time you run two coding agents at once it feels like free throughput. The second time, one of them commits a file the other was halfway through rewriting, and you spend the evening reconstructing what happened from a git reflog.

That is the whole lesson, and it arrives early: one agent is a tooling problem, several agents on one repository is a distributed-systems problem. Everything you already know about concurrency applies — mutual exclusion, liveness, failure detection, orphan cleanup — except the workers are non-deterministic, can hang for reasons that never repeat, and will confidently report success for work they did not do.

What follows is the set of failures I actually hit running a fleet against a multi-repository workspace, and the coordination model in Hilum Tools that removes them. None of it needs a supervisor process. All of it can be built into an existing setup incrementally.

Failure 1 — two workers claim the same slice

The naive dispatcher hands out a task list and trusts everyone to stay in their lane. This works until two tasks touch one file, or until a worker finishes early, looks for more work, and picks up something already in progress.

The fix is not politeness, it is arbitration at the point of hand-out. There is exactly one place work becomes owned — a claim_next operation that atomically selects an unclaimed item, marks it owned by a worker id, and returns it. Never "here is the list, pick one". A worker that wants work asks for work and is told what it got.

Two properties matter beyond the atomicity:

A claim carries a lease, not a lock. A worker that dies holding a permanent lock stalls the queue forever. A lease expires, the item returns to the pool, and someone else picks it up. The worker that comes back from the dead finds its claim gone and stops — which is the correct behaviour, not a bug to be worked around.
The claim states the blast radius. Which paths, which repository, which package. That is what makes the next failure preventable rather than merely detectable.

Failure 2 — parallel edits corrupt each other

Two agents editing one working tree is not concurrency, it is data loss with extra steps. The failure modes are specific and all of them are ugly: one worker's git add -A sweeps up another's half-written file; a formatter hook runs across a module that is mid-rewrite and fails, blocking a commit that had nothing to do with it; a build that one agent needs green is red because another agent is three files into a refactor.

Isolation is the only real answer, and the cheapest form of it is already in git: give each worker its own worktree. Same repository, same object store, separate checkout and separate index. Parallel edits cannot see each other, hooks run against a coherent tree, and the merge happens once, deliberately, at the end.

It is not free — a few hundred milliseconds and some disk per worker — so it is worth being precise about when you need it. Overlapping paths, shared build artifacts, or anything that runs commit hooks: isolate. Genuinely disjoint work in separate repositories: do not bother.

And regardless of isolation, one rule earns its place in every agent instruction file I maintain: never git add -A while another worker is live in the same tree. Stage explicit paths. This single line has prevented more incidents than any amount of clever tooling.

Failure 3 — output dies with the process

A worker finishes, writes its result to stdout, and exits. The parent was busy, or restarted, or the pipe buffer filled. The work happened; the record of it did not. You now have a repository in a state nobody can explain and no artifact describing why.

Treat worker output as durable state, not as a stream. Write it to a location that outlives the process — a file, a queue, a table — as it is produced, not at the end. The reader then polls that location and is free to be absent, slow, or restarted without losing anything.

The same applies to handoff. When agent A's work becomes agent B's input, that transfer should not be "A tells B" — a message in flight when either side dies is a message lost. It should be "A writes to a durable mailbox, B reads from it when it is ready". B picks up exactly where A stopped, with no re-briefing, and neither one needs the other to be alive at the same moment.

Failure 4 — the silent hang

This is the expensive one, because it costs wall-clock rather than correctness, and because you often do not notice for hours.

An agent waits on a container that will never start, a prompt that will never be answered, a network call with no timeout. It is not crashed — crashed is easy, crashed produces an exit code. It is alive, idle, and producing nothing, and every naive liveness check says it is fine.

Two mechanisms, in this order:

Event-driven completion is primary. When a worker finishes, it says so, and that notification wakes the orchestrator. This is the normal path and it should carry essentially all of the traffic.

A watchdog is the backstop. A timer set when work is dispatched, long — tens of minutes, not seconds — whose only job is to ask "is this still alive?" if no completion event has arrived. Long intervals matter for a reason people underestimate: a short polling loop is not merely wasteful, it also fails to detect the hang. Polling tells you the process exists. It does not tell you the process is doing anything.

For "doing anything", check ground truth instead of process state: is the working tree changing, is a compiler running, has a commit landed since the last check? A worker that has produced no filesystem movement and no child-process activity for N minutes is hung, regardless of what its status field says.

Then, having detected it: kill the process group, not the process. A stopped parent with running children is the worst of all states — the orchestrator believes the slot is free, dispatches into it, and now two workers really are in the same tree. And after the kill, recover rather than discard: hung agents have usually staged real, verified work that just needs committing.

The minimal protocol

Everything above collapses to five rules that can be added to an existing setup one at a time, roughly in order of return:

One place work becomes owned. Atomic claim with a lease and a stated blast radius. No "pick from the list".
Isolate anything with overlapping paths. A worktree per worker; stage explicit paths, never -A.
Durable output and durable handoff. Results and messages survive the process that produced them.
Event-driven completion, long-interval watchdog. Liveness checked against ground truth — tree movement, child processes, commits — not against a status field.
Verify independently before believing. A worker reporting success is a claim, not evidence. Re-run the check yourself.

That last one is not a distributed-systems rule, it is an agent rule, and it is the one that catches the most. Self-reported success is the single least reliable signal in the system — not because the agent is lying, but because "I ran the tests" and "the tests passed" are different statements that language models conflate under pressure. Anything load-bearing gets re-verified by whoever is going to act on it.

What not to parallelise

Throughput is not the only axis, and some work is simply cheaper serially:

Anything touching the same module. Merge conflict resolution between two agents costs more than the sequential run saved.
Anything writing to a shared index or lockfile. Two pnpm install runs against one store, two indexers writing one database — the contention costs more than the concurrency wins.
The first pass of unfamiliar work. Run one agent, see how it fails, encode the lesson, then parallelise. Fanning out before you know the failure mode multiplies the failure rather than the output.

Takeaways

One agent is tooling; several on one repository is distributed systems. Treat it that way from the first parallel run.
Ownership is arbitrated where work is handed out, with a lease — never by convention.
Isolate overlapping work in separate worktrees; stage explicit paths.
Output and handoff must outlive the processes that created them.
Completion is event-driven; the watchdog is a long-interval backstop; liveness is measured against ground truth.
Kill the process group, then recover the staged work rather than discarding it.
Never act on a self-report you have not independently verified.

The other half of making a fleet cheap is making each agent stop re-learning the repository: persistent memory for what it should already know, and token-budgeted retrieval for what it has to look up. If you want an orchestration layer designed for your own stack, that is the AI development engagement.