Lessons learned from building a software factory
On this page
In October 2026 we merged 2,901 merge requests into our main development branch. In December 2024 it was 39. September 2026 was 1,199.
The team stayed the same size, about five engineers. What changed is how we work with agents. This entry is the written version of a talk I gave at AWS Community Day in Ho Chi Minh City on 31 October 2026, including what broke.
Talk first, code last
Our main flow does not start with “write code”.
- Talk. We discuss an issue or a better direction with an agent, or ask it to research how to improve a feature.
- Decide. If there is a decision, the agent writes one HTML page with the options. The team reviews that page.
- Ticket. What we agree on becomes tickets.
- Build. Only then do agents write the code.
Human attention goes to the decision, not to reading every diff.
How we prompt
How we talk to agents turned out to be the main skill. Four habits:
- A discussion with only humans has already failed. Agents are capable; the slow part is us. So an agent is in every discussion, researching and drafting while we talk.
- Prompt for divergence, not convergence. Early on, ask for many directions and options, not one quick answer. Narrowing down is the human’s job, on the decision page.
- Give the goal, not the steps. Say what done looks like and let the agent find the path.
- Ask for a page, not a reply. A decision comes back as one HTML page the whole team can read and agree on.
One folder runs everything
We stopped choosing which folder, and which machine, to start an agent in. Every session now starts in one folder we call control. It holds the agent’s notes, our skills and a short guide for each machine. From there the agent reaches whatever the task needs: the app, infra and docs repositories on the laptop, our team machines, and the dev and production clusters in the cloud.
The notes stay in one place, so the next session starts with what the last one learned.
Setting it up is simple. Create an empty folder, then ask the agent to take notes there and maintain them as markdown files with frontmatter. The frontmatter (a date, a status, the systems a note is about) lets the agent find a note again and mark it stale when its facts change. This is the same idea as the file-based memory in agentic memory, entry 2, used for our own work.
Agents merge their own work
In October, 2,739 of the 2,901 merges, 94%, had no human reviewer assigned.
631 of them came from background agents with no person in the loop at all:
- 266 from scheduled jobs that run about every six hours and look for refactors, performance fixes and security issues.
- 365 from other background runs, such as overnight documentation and cleanup passes.
What replaces the human reviewer is proof. Before it can merge, every merge request must:
- start the full app,
- capture screenshots of the change,
- record a video of it working.
No proof, no merge.
The agent puts the proof into the merge request itself. For a small UI fix, that is the screen before and after, in light and dark mode, at a wide and a narrow width. Anyone on the team can see the result in a few seconds without checking out the branch or running the app.
Humans decide, agents execute
| People | Agents |
|---|---|
| Choose the direction | Research the options |
| Agree on the decision page | Write code and tests |
| Approve risky work | Merge small, safe changes |
Because the decision is made before any code, the merge request itself is fast: in October the median was 32 minutes from open to merge.
Small merge requests made auto-merge safe
At first we reviewed every merge request the agents produced. Review became the queue.
Then we split the work into small merge requests, small enough that the agent can judge how risky its own change is. We have let agents merge alone for several months with no incident caused by it. Only 2 of October’s 2,901 merges were reverts.
We did not make the agent smarter to trust it. We made the merge requests smaller.
What broke: the machine, not the model
Running more agents did not hit an AI limit first. It hit the machine. Every coding session ran the same tests again, on the same box.
So all sessions on a machine now share one test cache. It stores each passing test with a hash of the files that test used. Before a test runs, the session asks the cache whether this test already passed on exactly this code. If yes, it reuses the result. If the code changed, the test runs once and the result is saved for every other session. A warm-up job refills the cache after each update of the development branch, and 2% of hits are re-run at random to check the cache is not lying.
We did not trust it on day one. The cache started in shadow mode, comparing its answer with a real run, and only switched itself on after 3,000 confirmed hits.
- On the busiest cache day, 29 October, 72% of test results came from the cache: about 32,000 of 44,000.
- Over the last 7 days the average was 47%.
- 3,006 random re-checks found no wrong result.
We did the same for the other checks: type checks, lint, migrations, secret scan, dependency audit and tests, 65 in all, picked by what changed. A shared check runner keeps 10 warm copies of the app ready, so checks start at once. The median wait for checks went from 302 seconds to 73, 76% shorter.
Start small
If you want to try this next week:
- Pick one repeatable task.
- Agree before you build.
- Keep each change small.
- Turn every failure into a rule.
The next entries in this journey will go deeper into each part, starting with what the agents still get wrong.