← Back to Issue

Notes on the software factory

From Leadership in Tech · subscribed via aiste.ulozaite@gmail.com · original ↗ · unsubscribe

AI coding agents work best when given real tools like browser access, deployment pipelines, and test runners. Adding a critic agent to review work improves quality, and pairing that with more thinking time on hard problems helps further. Memory and long-running workflows are still weak points. The whole stack, from cloud infrastructure to IDEs, needs to be rebuilt around agents rather than adapted from human-centered tools.

{“@context”:”https://schema.org”,”@type”:”BlogPosting”,”headline”:”Notes on the Software Factory”,”description”:”Notes from a talk on vibe coding: giving agents tools, speeding up the bottleneck, critics, memory, skills, integration tests, and the agent native stack.”,”datePublished”:”2026-07-08”,”author”:{“@type”:”Person”,”name”:”Benedict Brady”,”url”:”https://benedict.dev”},”publisher”:{“@type”:”Person”,”name”:”Benedict Brady”},”mainEntityOfPage”:{“@type”:”WebPage”,”@id”:”https://benedict.dev/software-factory”}}

← Back

Notes on the Software Factory

These notes are adapted from a talk I gave at the Ellipsis Labs office about vibe coding and the software factory.

July 8, 2026

Give the Model Tools

The number one piece of advice is always the same: give the models tools to check their own work. That said, they are becoming much better at this naturally.

To the extent you can, you want to do things like this:

• Give the agent CLI access to deployments so that it can deploy and check for bugs
• Give the agent browser access for frontend (I think this comes out of the box now)
• Give the agent powerful machines to run tests/simulations etc. (I use Modal for this)
• Give the agent a Solana wallet with real money in it

Agent + Tools

AgentDeploy CLIship and check for bugsBrowsersee the frontendBig Computetests and sims on ModalSolana Walletyes, with real money

Give the agent the same toolkit a software engineer has

The goal is to complete the longest-running tasks, and this often requires them to have the tools that software developers have. Often there is one person on the team who sets this up correctly and can export their setup to the rest of the team.

There is a broader notion here that the interesting businesses to build are in the bottleneck space. Build a lab where you can parallelize experiments. Buy a lot of robotic arms.

Speed Up the Bottleneck

Identify what part of the workflow is slow and try to make it faster. This is a top annoyance for me. I often use loops for this purpose. I’ll kick off a monitoring loop that checks in on the current run and makes sure things are going smoothly.

Make Thinking the Bottleneck

BadBetterBestThinkingEverything else

Everything around the thinking step should be fast and parallelizable

Add a Critic

An idea that I have been partial to for a while is that the agent loop needs a critic to validate the result. Often this task is undertaken by the human monitoring the loop, but there are ideas around putting an agent in the loop as the critic. The critic’s clean context allows it to look more objectively at the work being done and propose both meta improvements and new directions.

The Critic Loop

Critic Agentvalidate the result,not just the diffimprove theprocess itself

A clean context watches from a distance and improves the process

The common riff on this idea is /goal. Goal is used to define a rubric in text space. Then the model iterates in a loop until the goal is deemed complete. This solves some of the problems around long-horizon coherency.

Quality Scales with Thinking Effort

There is now pretty robust data that benchmark scores scale with thinking time. This means you should really toggle the thinking up for harder problems.

Measure Twice, Cut Once

The common conventional wisdom used to be to add a planning step explicitly. A lot of the mechanics around how planning was implemented were related to context window limitations that have started to ease. Nowadays I would consider just skipping plan mode but keeping the principle in mind.

Plan Mode

Plan firstplanone clean buildDive in…

Keep the principle even if you skip the mode

Workflows and Subagents

The models have become quite smart orchestrators of the lower-level models. This means that we are now able to speed up work a lot by having one higher-level model manage a bunch of lower ones. This doesn’t impact the overall model intelligence much, but it does impact the speed at which tasks get completed. Dynamic workflows are a riff on subagents where the task list is specified as a DAG. This seems to be quite elegant and flexible but consumes a ton of tokens.

Orchestration

Big Modelplans, delegates, reviewssmall modelsmall modelsmall modelsmall modelsmall model

One big model managing many small ones

Memory Is Weird but Improving

The ideal version of memory would be something that runs in the background silently and learns your preferences. In reality the harness doesn’t do this well yet. It will write or delete things from the memory file, but it is not trained well on the wide range of human preferences. I assume this will be solved eventually along with a host of other continual learning ideas.

Memory Across Sessions

What it knowsSessionsdeleted the wrong thingidealtoday

The gap between ideal memory and what the model does today

There are ideas around dreaming and reflecting that may make the memory stack more reliable.

Skills vs. The Bitter Lesson

One way to think about skills is as documentation of a workflow or idiosyncrasy of your setup vs. the baseline. At this point the models are basically smart enough to figure anything out, but it is frustrating to have them relearn everything from scratch every session. So skills abbreviate this process since the models are trained to look at them first. As the models get bigger and smarter I would expect skills to fall away.

Skills Skip the Ramp-Up

Contextproductivesession 1session 2session 3session 4with skillsfrom scratch

Skills start each session closer to productive

Moving Up the Abstraction Ladder

The current harnesses have too many bells and whistles. Given that the models are generally smart, we should expect the knobs to live one level below the UI and the models should just learn to change their own settings. In a perfect world, the harnesses actually get simpler as they get more advanced. This is related to the idea of the meta-harness, which you can read more about here.

Integration Tests

The correct way to validate changes in the new coding regime is by having a robust set of integration tests. This is how Bun and other teams have completed large buildouts in record time. Anything in your system that is not end-to-end testable is a massive liability. At Meridian, we rewrote our entire stack into modular TypeScript to make this process easier and review more streamlined.

The Integration Loop

agent writes codeintegration suiteship itfail: fix and retry, no human neededyour systemend to end testableliability

Anything not end-to-end testable is a liability

Replacing GitHub

If we want agents critiquing all PRs and full integration suites that our agents can easily run against, you can imagine why GitHub feels like a little bit of a relic. There is also some sense in which the output of code is kind of a compression of the work put into it. It would be nice if agents could communicate in a higher bandwidth way (traces, etc.). So many folks are thinking about what the next generation looks like here for enterprises.

Communication Bandwidth

Todayagentthe diffagentNextagentskills · traces · contextagent

Agents want to share more than a diff

The Agent Native Stack

This is basically true for all parts of the stack from AWS up through the IDE. I imagine we will end up with a fully vertically integrated cloud solution that is designed custom for agents and not humans. Some qualities this needs:

• Fast deployment loop
• Agent native traces
• Easy to test and roll back
• Tiered permissioning
• Alerting callbacks for agents

The Agent Native Stack

awsvercel + railwayapp layeragent plugins

Is Typing Too Slow?

Typing is a pretty low bandwidth way to communicate ideas. It is hard to imagine that we will still be typing into our computers in a few years. The intermediate solution is voice-to-text transcription, but the future probably looks more like a fluid conversation with an assistant, an always-on microphone capturing casual conversations for context, and some sort of live video input. The Thinking Machines interaction models are a nice take on this idea.

Input Bandwidth

todaythe next generationTypingVoiceConversation CaptureLive Video

Voice, ambient capture, and video carry more context than typing

Local vs. Cloud

One of the biggest debates around the dev stack is whether the agent should be primarily housed on a local machine or in the cloud. It is much easier to get started on a local machine and it is easier to model the permissions of the agent. But as many have noticed, local development does not scale well and is not natively multiplayer. For this reason, cloud has been on the rise recently and many products are ending up in an awkward spot, split between the two.

Local vs. Cloud

LocalCloudproductsawkwardly split

Current Weaknesses

• Extremely long-running workflows are hard to manage
• Extracting business context is painful
• Continual learning / sample efficiency
• Computer use / restricted platforms
• Always-on agents is unsolved