The pitch for autonomous coding agents is easy to write and hard to deliver. Point a model at an issue tracker, let it open merge requests, watch the backlog drain. In practice the first version of anything like this produces a lot of confident merge requests that do not do what the issue asked.
We have been running a small agent team against our own GitLab instance for the better part of a year — a PM agent, a Dev agent, and a QA agent, each exposed over the Model Context Protocol, with a GitLab server providing the shared substrate they all read and write. This is a note on the parts that turned it from a demo into something we let touch real branches.
The shape of the system
Three agents, one repository, no shared memory between them.
- PM agent takes an incoming request, decides whether it is a real unit of work, and writes a planning issue. It owns triage, assignment, and escalation.
- Dev agent reads a planning issue, implements against a branch, and opens a merge request. It also handles the second pass when QA sends work back.
- QA agent reviews the merge request against the acceptance criteria, scores it, and posts a structured verdict.
Everything the agents know about each other arrives through GitLab: issues, comments, labels, MR diffs, pipeline results. There is no orchestrator holding state in memory, no message bus, no shared context window. If a run dies halfway through, the next invocation reconstructs everything it needs from the issue thread. This was not the original design. It is the design that survived.
Contract negotiation before implementation
The single change that improved output quality most was making the agents agree on what “done” means before any code gets written.
The Dev agent’s first action on a planning issue is not to implement. It is to read the issue and post a proposed set of acceptance criteria as a comment. The QA agent then reviews that proposal — for specificity, testability, completeness, and whether it actually matches the scope of the issue — and replies APPROVED or REJECTED with reasons. Only after approval does implementation start.
This feels like bureaucracy until you watch what it catches. A vague issue like “improve the export performance” produces a criteria proposal that immediately exposes the ambiguity: improve by how much, measured how, on what dataset. QA rejects it, the proposal gets sharpened, and the implementation that follows is aimed at something checkable. Without the gate, the Dev agent would have picked an interpretation silently and the disagreement would surface at review time, after the work was done.
The negotiation also gives the QA agent something concrete to review against. Reviewing a diff against “the issue” is a judgment call. Reviewing it against five numbered criteria that both agents already agreed to is closer to a test.
Structured scoring instead of prose review
Early QA output was a paragraph of commentary ending in an implied verdict. It read well and was nearly useless as a control signal — you could not automate on it, and the Dev agent’s second pass would fix the things mentioned most emphatically rather than the things that mattered most.
The review is now a scored evaluation: each acceptance criterion gets a pass/fail with evidence, plus separate dimensions for correctness, security, and test coverage. The aggregate determines whether the issue moves forward or gets labeled for rework. Labels drive the state machine, so the score is not advisory — it is the thing that decides what happens next.
The side effect worth noting: structured output made the QA agent measurably stricter. When you have to attach evidence to each criterion, “looks fine” stops being an available answer.
Retry loops that carry the reason forward
When QA fails a merge request, the Dev agent does not start over. It reads the latest evaluation comment, implements fixes against the existing branch, pushes to the existing MR, and posts an update comment describing what it changed and why.
Three details matter here:
- The failure reason travels with the retry. The agent reads the specific failed criteria, not a generic “try again.”
- The MR is the unit of continuity. Reviewers — human or agent — see the history of attempts on one thread instead of a graveyard of abandoned branches.
- The update comment closes the loop. QA’s next pass reads what the Dev agent claims to have fixed and verifies exactly that, rather than re-reviewing everything from scratch.
Retries are capped. An MR that fails three rounds gets escalated to a human rather than looping until the token budget runs out. The escalation path is the most important part of the retry design and the easiest to forget to build.
What we would tell someone starting this
Use your existing systems as the state store. The temptation is to build an orchestration layer with its own database. GitLab already has durable state, permissions, audit history, and a UI humans understand. Writing agent state into issues and labels means a human can inspect, correct, or take over at any point in the workflow without special tooling.
Separate the agents by incentive, not by task. The value of a QA agent is not that it does a different kind of work — it is that it has not already convinced itself the implementation is correct. Give it its own prompt, its own context, and no visibility into the Dev agent’s reasoning. If both agents share a context window, you have one agent talking to itself.
Make disagreement cheap and visible. The criteria negotiation and the scored review both exist to surface disagreement early, in writing, where a human can read it. An agent system that never disagrees with itself is not aligned; it is just agreeable.
Budget for the boring parts. Contract negotiation, structured scoring, retry state, and escalation are most of the engineering. The model calls are the easy part.
The system does not run unattended and we do not want it to. What it does is take work from “someone needs to think about this” to “someone needs to approve this” — and that turns out to be most of the distance.
Sophron Labs 