I’m very happy to announce that, together with Maven, I’m launching a self-paced cohort.

The course, on your schedule

If the timing or pricing of the live cohorts didn’t work for you, I hope this opens the door:

• Start anytime. Get instant access when you enrol.

• Set your own pace. Fit the lessons around work and life. Pause, practise and revisit the material whenever you need to.

• Introductory price: US$650 through November 2026.
  That’s exactly half the price of the original live cohort.

Enrol in the self-paced cohort on Maven →

Agentic Engineering is now part of the job

Over the past year, agentic engineering and AI-assisted software development have moved from experimentation into mainstream practice.

• AI is part of everyday development. In Stack Overflow’s 2025 survey, 51% of professional developers already reported using AI tools daily.

• Agents are doing real engineering work. They’re contributing to production workflows, from implementation and testing to debugging and documentation.

• The skills extend across roles. Developers need dependable workflows. Engineering leaders need repeatable team practices. Product professionals have new opportunities to build and contribute directly.

For everyone involved in building software, upskilling is now a must. This is the time to develop a solid understanding of context, specifications, delegation and verification.

What you’ll learn

This is the most comprehensive course on the market for professional AI-assisted software development.

1. Context engineering. Build a portable context stack from documentation, architecture, conventions, rules and agent skills. Give agents the understanding they need to work effectively in your project and across a team.

2. Specification-driven development. Turn ideas into clear briefs with intent, constraints and acceptance criteria. Use specificity and planning to get more dependable results and reduce rework.

3. Agentic software engineering. Explore coding modalities, delegation, sub-agents, orchestration and harness engineering. Learn how to choose the right level of autonomy and stay in control.

4. AI across the development lifecycle. Apply these practices to features, refactoring, migrations, code review, testing, documentation, pull requests and DevOps.

5. Agents at scale. Work with background and asynchronous agents, parallel tasks, autonomous execution and Continuous AI. Cover security, sandboxing, secrets, measurement and enterprise adoption.

What’s included

• Video lessons and the course reader. A substantial body of teaching and reference material to work through on your own schedule.

• Demonstrations and homework walkthroughs. See the techniques in practice, then apply them yourself. Homework includes fundamentals and more ambitious stretch goals.

• The private Discord community. Ask questions and get professional help with any challenge, share your experience, and discuss real projects with me and other course participants.

• Lifetime access. Return to course content, recordings and the community as your work develops.

• A certificate of completion. Share your new skills with your employer or on LinkedIn.

Bring your tools and your codebase

• Use your preferred AI tool. Claude Code, Codex, GitHub Copilot, Cursor, OpenCode and others. The course teaches transferable principles.

• Work in your own language. Examples and homework include Python and TypeScript. You can complete assignments in your own language and codebase.

• No prior AI or machine learning training required. The focus is practical software development.

Built on successful live cohorts

Five-star feedback from students, with a world-class NPS.

The live cohorts have brought together developers, engineering leaders and other software professionals. Their feedback highlights:

• Immediate application. Putting the lessons to work in existing projects and workflows.

• Technical depth. Developing a more structured approach to working with agents.

• Greater confidence and effectiveness. Understanding how to guide the tools and assess their output.

Read more about the course and feedback from previous students →

Join when you’re ready

Thank you to everyone who asked for a self-paced option. I’m excited to welcome more of you to the course.

Join the self-paced cohort on Maven →

Start anytime. Learn at your own pace. Introductory price: US$650 through November 2026.

The Warp team uses it to build Warp Terminal and Warp Factories itself. Ben started with a whiteboard, then opened the team’s working dashboards and examples. The demos covered how Warp routes work to agents, brings the results back for human review and improves the factory over time. They included scoring rubrics, proposed changes to instructions and benchmarks built from the team’s own tasks.

Hugo Bowne-Anderson and Eleanor Berger teach Build Your Agentic Software Factory: a practical course on directing agents, checking their work and turning an idea into a working application.

One Factory the Team Can Improve Together

The architecture Ben sketched starts with a foreman: the agent the team talks to when it wants to put work into the factory. It coordinates what happens next, doing a task itself or delegating parts to other agents. Warp’s example adds triage, implementation and code review. A request can move through research and a specification, implementation, then a few rounds of review before a pull request reaches a person.

This gives the team a single place to agree on the skills, integrations, permissions and model choices those agents use. Improvements to that setup can then reach everyone’s work. There is a clear place to investigate when the factory keeps making the same mistake.

At the time of the talk, the whiteboard assigned Opus 5.5 to the foreman and implementation, GLM 5.3 Flash to triage, and GPT-6 Sol to code review. Ben presented this as an arrangement to test against the team’s tasks. It illustrates using a cheaper model for research and a different model family to review the code. The model names will change; the decision is how to spend the team’s budget across the work the agents need to do.

Warp Factories defines the setup in code, including YAML configuration, so changes can go through version control and review. The dashboard presents that configuration and the runs it produces. The factory is something the team can change deliberately and test.

Give Agents the Environment Their Work Requires

A shared factory still needs somewhere to run its agents. Ben compared two approaches: a persistent server that receives the team’s work, perhaps with containers inside it, and separate environments started for individual agents.

Ben described a persistent-server pattern he had seen with OpenClaw and Hermes, sometimes with containers inside it. It avoids waiting for a machine to start and can run on hardware the team owns. Isolation between tasks needs attention, and one operating system may not suit every job.

Warp had a concrete reason to separate environments: its factory mostly ran on Linux, but work on an iPhone app needed macOS. The orchestration could stay on Linux while an implementation agent used a macOS runner.

Separate environments introduce deployment complexity and startup delays. The choice follows from the work: the operating systems, task isolation and response times the team needs. For a team of one or a few, Ben also suggested that a Mac mini tucked away in a cupboard could be enough.

Start with Background Tasks, Then Work with the Factory

Ben suggested introducing a factory gradually, starting with work that already has a trigger: investigating a CI failure, looking into an error alert, or checking the unassigned backlog each morning. Each automation connects a trigger to an agent run and an output, such as a pull request or a message.

This gives the team something to inspect while people continue using their existing coding tools. As confidence grows, the factory can take on more of the work people actively direct.

At Warp, Ben showed that interaction happening in shared Slack threads. Engineers can ask the agent to do work, discuss the result and see the same conversation. The factory also reports the cost of those interactions.

The iPhone example made this concrete. An agent built the app on a macOS runner, started a simulator and returned screenshots. Ben could inspect the interface and request changes without pulling the branch onto his laptop or starting Xcode himself. Once the visual result looked right, he could review the GitHub pull request.

The agent’s evidence becomes part of the review. A screenshot can answer a question about the interface; other checks are needed for behaviour it cannot show. The factory needs to return enough evidence for the person directing the work to judge the result.

Score Runs to Find Repeated Problems

Once work is flowing through the factory, the team needs to know how well it is running. Ben showed a dashboard that tracks productivity, spending and cost per pull request. It separates inference costs, the model calls billed in tokens, from compute costs, the machines running the agents’ environments.

Those totals tell the team where to look. They do not explain why a run took too long or needed several rounds of correction. Ben described how Warp investigates those problems with scoring agents: models that assess completed runs against written criteria.

Ben showed rubrics for efficiency and code quality. The code-quality configuration distinguishes work a reviewer would accept, work with insufficient evidence to judge, and work with a defect that needs fixing. Its instructions ask the scorer to evaluate the code diff against the repository’s standards and hold back when there is too little code to judge.

In Ben’s examples, efficiency scoring can expose problems even when the agent eventually finishes. Did it keep making failed tool calls? Did unclear testing instructions force it to work out how the tools were supposed to behave? How much human intervention did it need?

These are judgements against the team’s rubric. They give the team a way to find runs worth examining, alongside the code, tool calls and review history. The rubric itself needs care: if it rewards the wrong behaviour, its scores will point the factory in the wrong direction.

Turn Failures into Changes People Can Review

Warp’s self-improvement agent looks across scored runs for recurring problems and proposes changes. In response to an attendee’s question, Ben explained the timing: a scorer assesses completed work, while a separate scheduled job reads recent failures, their scores, notes and full conversations. He said once a day was probably the team’s current default for that improvement pass.

One example he opened was a draft pull request to improve branch-handling instructions: check whether a remote branch name already exists before pushing. Its description linked back to runs that prompted the proposal.

This was a proposed improvement. Ben did not show a measured saving from merging it. The example showed how the factory can turn a recurring issue into a reviewable change, with evidence a person can follow.

Ben also described a broader problem from his own setup: poorly written testing instructions and a troublesome test harness made agents waste time on failed calls and workarounds. Looking across conversations helped an improvement agent identify that pattern. An individual task could finish while the underlying problem kept adding cost to later tasks.

The changes here are to the factory’s instructions, skills and configuration. This is a feedback loop around the system running the agents. People still decide whether to accept the proposed changes.

Crediting Warp CEO Zach Lloyd’s phrasing, Ben called this factory engineering: improving the system the team relies on to build software. Product engineering continues to involve deciding what the product should do and how people should experience it. Factory engineering examines the cost, speed and quality of the process producing it.

Benchmark the Team’s Own Work

Ben’s final demonstration tackled a question that comes up whenever a new model arrives: would it improve this factory?

Warp builds benchmarks from its own completed tasks. An agent helps identify candidate conversations and estimate their difficulty, then a person narrows the selection. Each benchmark task records the repository state to start from, a prompt and success criteria. The factory can replay that work with a different configuration and score the result.

The chart Ben showed compared configurations on judge-rated correctness and average API cost. Each point represents a configuration tested on those tasks. He also switched the view to latency: the time needed to do the work. Those comparisons can inform a choice when a team cares about lower spending, faster responses or the strongest results on its selected tasks.

Ben was explicit that the results were specific to Warp’s setup and sample. A team building an iOS app may get different results from one doing front-end work or a specialised migration. The scores also depend on the success criteria and judge used in the benchmark.

Ben explained how to keep an experiment focused by changing only the implementation agent’s model. The other agents’ models stay fixed, so the team can compare the implementation model’s performance within the same system. Warp was also investigating whether different agent arrangements worked better: one agent doing everything, a foreman with a reviewer, or a larger assembly line. Ben did not present a settled winner among those arrangements.

Running these comparisons costs money. Ben noted that the team can reduce the number of candidate models, tasks or repeated runs to make the experiment manageable. The purpose is to get evidence for a decision the team needs to make.

Keep Discussion and Ownership Clear

In the Q&A, Eleanor asked how Ben handles work that can run in parallel alongside tasks that depend on one another. Ben tends to start with one implementation thread, then split emerging work into well-scoped issues. Those issues can feed an automation or become separate conversations.

That preserves an opportunity to discuss each piece. Ben was experimenting with ways for agents to recognise when a task needs further discussion before implementation. Delegating more work can also mean delegating more assumptions, so those questions matter.

Eleanor described planning the dependencies with an agent first, then using blocking relationships in Linear to tell the factory which tasks could start. Both approaches make the relationships between tasks explicit enough to direct the work.

Ownership stays with people too. Ben said Warp’s agents open pull requests under their own identity and include the source conversation and initiating person. For directly scoped work, that person reviews and merges; larger changes can involve another human reviewer.

Warp’s approach makes the factory’s operation available for the team to inspect and improve: what the agents are allowed to do, what evidence they return, which failures recur and whether a change improves the work. That gives the team a way to develop the factory alongside the product.

More information is available on the Warp Factories website, and the full session is available on Maven.

Build Your Agentic Software Factory on Maven

Hugo Bowne-Anderson and Eleanor Berger teach Build Your Agentic Software Factory on Maven, a practical course for builders who want to put these ideas into practice.

Over two weeks, participants set up a factory locally or in the cloud, give agents clear outcomes, context and constraints, and define the checks they need before trusting the agents’ work. They start with a conversational utility, extend the approach into a working web application, and learn how to publish or share what they have built.

The course is for builders with or without coding experience, working on ideas from their own work or life. Live workshops, office hours and project feedback help participants build a repeatable way to delegate, inspect results and keep improving their factory.

👉 Course details and enrolment on Maven.

The answers come in three flavors – choice, noul, and score – which I’ll describe below.

The utility of this model comes from its speed and the accuracy with which it makes its numerical predictions. Conventional LLMs are orders of magnitude slower, and if you ask how confident they are, they’ll give you a number that often has little bearing on reality.

The demos for Jev are super impressive. In their Doom demo, they send 7 requests per second, having Jev predict which button should be pressed in real time. Jev can easily navigate the Doom map and blow away the baddies. ... Try that with an LLM.

The API in brief

A request has a state and a questions map. The state is like the prompt in a conventional LLM: all the text or JSON data you want the model to consider. Each named question in questions has a type, instructions, and sometimes a set of allowed answers or a rubric.

Here are the three question types:

• choice: picks one item from a fixed list. It returns {choice, probabilities, confidence}.

• score: picks a position on an ordered rubric. It returns {score, legend, probabilities, confidence}.

• noul: asks whether something is true. It returns {noul}, which is the probability from zero to one that the answer is yes. No separate confidence field on this one.

Example: handicapping a horse race

For example, the input for a horse race could look like this:

{
  "model": "jev-latest",
  "state": {
    "race_details": {"distance_m": 1600, "track": "muddy"},
    "horse_details": {
      "thunderhoof": "Won 2 of last 3 races; strong in mud.",
      "saddleflash": "Won last race; prefers dry tracks.",
      "gallopjack": "Second in last 2 races; untested in mud."
    }
  },
  "questions": {
    "winner": {
      "type": "choice",
      "instructions": "Predict the winner of this horse race using horse ability, recent form, and fit for the course conditions. Select the horse most likely to win, not merely the favorite.",
      "criteria": {
        "thunderhoof": "Thunderhoof wins the race.",
        "saddleflash": "Saddleflash wins the race.",
        "gallopjack": "Gallopjack wins the race."
      }
    },
    "thunderhoof_handles_mud": {
      "type": "noul",
      "instructions": "Does Thunderhoof appear likely to perform well on the current track condition?"
    },
    "race_competitiveness": {
      "type": "score",
      "instructions": "How competitive is this race?",
      "criteria": [
        "One horse is clearly favored.",
        "A few horses have plausible winning chances.",
        "Many horses have similarly plausible winning chances."
      ]
    }
  }
}

And the answers would come back under those named questions (illustrative response – I made up these numbers):

{
  "answers": {
    "winner": {
      "type": "choice",
      "choice": "thunderhoof",
      "probabilities": {
        "thunderhoof": 0.48,
        "saddleflash": 0.27,
        "gallopjack": 0.25
      },
      "confidence": 0.71
    },
    "thunderhoof_handles_mud": {
      "type": "noul",
      "noul": 0.82
    },
    "race_competitiveness": {
      "type": "score",
      "score": 1.61,
      "legend": {
        "0": "One horse is clearly favored.",
        "1": "A few horses have plausible winning chances.",
        "2": "Many horses have similarly plausible winning chances."
      },
      "probabilities": {
        "0": 0.04,
        "1": 0.31,
        "2": 0.65
      },
      "confidence": 0.63
    }
  }
}

What’s under the hood?

How do they do this? My initial guess was that this might be a new take on BERT – Take a transformer encoder, pull in all the text and process it like a normal, and replaced the output head with something that has a bunch of “slots” of these different types. The API’s questions are associated with slots that select among a set of choices, score a rubric, or produce the probability that a statement is true.

Then, internally this gets turned into a prompt that dumps in the request structure (state and questions) but replaces the actual variable names with slot names so that the model knows where to stick the output data. Then, once the answer comes back, the API translates the slot names back to variable names.

This X post provides an even simpler possibility. This person (Harsha Gundala) has apparently replicated some of the success of Jev by fine tuning a conventional LLM. Rather than having a special output head, they just make the LLM generate the next token and then they use the logprobs to populate the probability numbers. For instance, if the question is boolean, then they look at the tokens true and false; if the question is over a set of enumerated choices, then they look at the logprobs of the tokens A, B, C, D, etc. Makes perfect sense.

Dang it... it makes perfect sense. I even wrote about a similar idea a year and a half ago. In that post I show how logprobs can be used as a “soft classifier” in exactly the way that the above X post says that Jev works. But I didn’t follow up on that approach because I was getting some wonky values for logprobs in some of my experiments. Shame I didn’t follow that to its logical conclusion, found a company, and pull in millions of dollars in venture capital! Oh well. At least I don’t have to think hard about the hero image for this blog post. I’ll borrow the one from my old blog post.

In any case, the secret sauce to make the outputs make sense is in training the model. Starting with a pre-trained model, they would likely follow up with supervised fine-tuning. The data set they use would be really interesting because they need to collect outcomes for events that happened in real life and make up random questions like these with known answers.

The weird challenge is coming up with good confidence or probability values for each training example. For instance, if you know Thunderhoof did win, you have a clean answer for the Choice question or the Noul question. But you don’t automatically have a clean answer for whether this particular input should have caused the model to be 51% confident or 95% confident.

Perhaps that’s where the reinforcement learning comes in – what TypeSafe is calling “Reinforcement Learning for Calibrated Decisions”. SFT would push the model to mimic the training data (where confidence is poorly defined), whereas reinforcement learning would allow the model to “introspect” into its own knowledge state and learn confidence/probability values that aren’t just wild guesses. For this, you would definitely need to collect data that was outside of the original pre-training and SFT set.

Another place where RL can be useful is in a domain like the Doom demo. Given some “environment” (which could be real-world 3D like in the Doom video, or it could be a board game, or anything), you can have the model make predictions about what will happen next and how to interact in the world. Then you can let that play out in simulated worlds and use the results for policy updates.

Finally, these models aren’t doing any reasoning. Reasoning requires token generation, which is much slower than processing input tokens and would slow these models down tremendously. And like I said, these are probably encoders only, so they wouldn’t be able to produce tokens in any case. Though I do wonder if they have some generic “thinking” placeholder tokens. There’s precedent for this. In Think before you speak: Training Language Models With Pause Tokens, they trained models with repeated <pause> tokens before the answer, giving them extra computational workspace without adding any new information. Something like that could be going on here.

What can you do with this?

I’ve been wondering how LLMs might be used for embodied AI. My concern has been that any conventional LLM that cranks out one token at a time is going to be way too slow to interact with the world and make decisions in real time. So, perhaps these embodied AI agents will use LLMs for high-level decision making and then use dynamic models of real-world systems that can be called as tools. For example, the LLM could say “catch the ball that the boy is throwing at you” and then call the catch_ball tool, which internally uses visual input and a kinematic/dynamic model of the robot to actually reach out and catch the thing.

But this feels awkward. The catch_ball tool is incredibly bespoke. You’d have to make a buzillion little tools like this, and even then it wouldn’t be great. How would you fold laundry? Instead, you’d want lower-level functions that generalize to any task, like move_left_hand_to_coordinates(x,y,z), but an LLM is going to take way too long to generate the tool call and won’t have the correct x,y,z values anyway.

This is where Jev might come in handy as an intermediate layer. Go watch that Doom demo again for a hint about how this might work. The high-level, abstract, navigational thinking could be performed by a conventional LLM, while the low-level motion of robotic limbs could be handled by conventional kinematic/dynamic models. In between, Jev could be watching streaming video from the robot’s camera and estimating the ball’s trajectory parameters to figure out the x,y,z location where the robot should place its hand to catch it.

(Update: The current model does not actually take visual inputs. Apparently, they’re feeding it text telemetry from the Doom environment. But since this is a transformer model there’s no reason that future models wouldn’t be able to process image inputs.)

What’s more – and it blows my mind to think about the possibilities – the high-level LLM can specify requests to Jev on the fly, as needed. The LLM, using Jev as a tool, can call on it to make arbitrary predictions about salient things in its environment. This is very generalizable!

There are plenty of other things you can do with a model like this:

• Document intake. Review, classify, and route incoming documents. For instance, reason through a stack of resumes to determine if a person is a good match to the job they are applying for.

• Spreadsheet AI computations. Quickly and cheaply classify text or estimate things like text sentiment across hundreds of rows.

• An agent or model router within your agent harness so that the best agent is selected for a particular task.

• Quickly read foreign skill files and foreign inputs to scan for prompt injection.

• Reading tool calls before submitting them to make sure they are safe. Easy example – scan for secrets.

• Image classification tasks in the future once the underlying model can process images. Stick it in smart glasses or an iPhone.

• Quickly classifying and sorting memories collected by an agent so they can be stored in the correct location.

• Fuzzy if statements. For example taking human interactions and determine whether to show them help text or offer refund or escalate to a human responder.

• Quickly review RAG search results for relevance to current context.

• Quick AI decisions, for instance useful in game engine to power the behavior of non-player characters where the number of possible decisions is limited.

• AI “salience” judgement. For proactive AI, use Jev to determine whether an incoming task should be dealt with now or put somewhere on a backlog.

• Just-in-time context gathering. Useful for a long-running, top-level orchestrator agent that is constantly context switching as you feed it every task you’re working with. (I keep a “generic agent” around for random tasks so that it has access to all the context I’ve been giving it, but it’s kinda dumb because the useless context stays with it and I pay for it!)

• Browser and computer usage where a traditional LLM decides the broad goal, Jev quickly decides the next legal actions to take, the browser or accessibility code actually do the text entry and button clicks.

• Super fast/cheap LLM-as-Judges.

• For AI search, one challenge you run into is making the search select the correct filters, sometimes from a very long list of possible filters (think about a complex hierarchical taxonomy of products for example). Jev could be given the entire list and instantly predict the most likely filters to try.

• Query understanding - I want to give ecommerce companies the ability to tag “ecommerce part of speech” for incomming queries. So instead of part of speech – noun, verb, adjective, you could identify which search terms are brand, color, style, product, etc.

• Aside detection – When chatting with an agent, have a hook that determines if the current question is in a side. If so The current conversation will be forked rather than muddling the context and disrupting the current work.

But... the key thing yet to be proven is how accurate it is in general real-world tasks. It might be the case that Jev – a general model – is not terribly accurate on your specific domain. Maybe the better approach is to just use TypeSafe’s process and fine tune a model for each specific domain.

But are those probabilities any good?

Let’s revisit some wonky values I ran into a couple of years ago, while experimenting with the my “soft classifier” described in my old blog post.

Let’s say you have a coin that comes up heads 60% of the time. If I ask you for the probability that the next flip comes up heads, any sane human should say 60%. Nothing tricky here.

But when I asked a conventional LLM (probably GPT-4) to answer “heads or tails” and looked at the probabilities derived from its logprobs, I saw something strange. If the coin’s heads rate was anything above 50%, the probability associated with the heads token would be almost 100%, while tails would be almost 0%. I wanted the odds of the next flip, but the model’s probabilities collapsed toward the winning answer.

So... does Jev do better?

We can ask it the same question in two different ways. A noul question asks for the probability that the next flip will be heads. A choice question asks it to pick heads or tails, and gives us probabilities for both. Here’s the request I made:

{
  "state": "We have an unfair coin that comes up heads 60.0% of the time.",
  "model": "jev-latest",
  "questions": {
    "next_flip": {
      "type": "choice",
      "instructions": "Which side will come up on the next flip of this coin?",
      "criteria": {
        "heads": "The next flip comes up heads",
        "tails": "The next flip comes up tails"
      }
    },
    "will_be_heads": {
      "type": "noul",
      "instructions": "The next flip of this coin will come up heads."
    }
  }
}

And here’s what came back. Unlike the horse-race example above, these numbers are real:

{
  "model": "jev-1.13.0",
  "answers": {
    "next_flip": {
      "type": "choice",
      "choice": "heads",
      "confidence": 0.99,
      "probabilities": {
        "tails": 0.01,
        "heads": 0.99
      }
    },
    "will_be_heads": {
      "type": "noul",
      "noul": 0.58
    }
  },
  "usage": {
    "input_tokens": 360,
    "output_tokens": 51
  }
}

The noul answer is almost what we’d hope for: 58% instead of 60%. But look at that choice response. It gives heads a 99% probability! Heads is the better bet, sure. But that doesn’t make the next flip 99% likely to be heads. We told it the odds were 60%.

Okay, let’s extend this across the full range of possible coin biases. I swept the stated heads rate from 0% to 100% in steps of 0.25 percentage points – 401 requests in all. The dashed line in each chart shows the ideal answer, where Jev’s probability matches the coin’s stated heads rate.

First, choice:

There’s that familiar pattern. Above roughly 50%, the heads probabilities jump to almost 100%. Below that, they tend toward 0%, with some messy behavior near the middle. As probabilities of the next flip, these values are very inconsistent with reality. They collapse to the winner.

Now compare that with noul:

That’s much closer! noul does a pretty good job of tracking the expected answer, although it generally undershoots. The biggest miss was 9 percentage points: when the coin’s heads rate was 49%, Jev said 40%. The 48% coin was tied for worst, with Jev returning 39%.

So this gives me a bit of pause. Asking for a probability with noul works reasonably well here – though still more off that it should be. Asking about the same event with choice gives me something that is just wrong. How will I know which kinds of questions Jev handles well, and which ones give me confident-looking numbers that don’t mean what I think they mean?

Should TypeSafe be scared?

Presuming my above coin toss example is a fluke and presuming the typical outputs are more accurate than this... if I was TypeSafe, I would be concerned about competing head to head with Anthropic and OpenAI.

As I covered above, Jev isn’t a conventional LLM - it can’t generate text. But its architecture is pretty close to one, it’s just being used differently. A conventional LLM iteratively calculates the probabilities for each token and then picks the highest, and those probabilities are typically hidden from us. Jev instead predicts only the next token and uses those probability values to calculate the answer. Over-simplified example: you give Jev “User comment: This product sucks!” and ask “Choose sentiment: satisfied, dissatisfied”. Jev ignores every token except those two and reads off “satisfied: 1%; dissatisfied: 99%”.

So, in principle, all that OpenAI or Anthropic might have to do is swap out the output head of the LLM with something fine-tuned to act like Jev and then they have their own competing product. And with the speed at which Jev clones are arriving, this might be an easy feat.

What’s more, the frontier labs would have an even more compelling product than Jev, because they could run both the conventional LLM and their version of Jev concurrently, in the same GPU! During a GPT-6 reasoning trace, the LLM could decide it’s time for a quick Jev decision, swap over to that head, emit a single token, and then jump back into conventional token generation after that. Frontier models would gain a new ability to make certain types of snap judgments more quickly.

The big unknown for me is still what Jev’s “Reinforcement Learning for Calibrated Decisions” actually is. If that’s their secret sauce, they better hold it close! Maybe generating datasets of “calibrated decisions” is genuinely hard. And maybe it’s difficult to coax a model into producing realistic probabilities when it’s partly learning from past events that don’t come with probabilities attached – either the event did or did not happen.

What I really want

What I really want next is a fine-tuning API for Jev where I can dump in a big blob of text and data, have TypeSafe convert all of that into good training data with example states, questions, and outputs, and then fine-tune the model for me. If TypeSafe offers anything like that, then I’m going to gather up all the horse racing data I can find and head to the tracks. 😆

That opens up a rather interesting question: what would you build if you could put a team of AI agents to work?

and I are launching Factory Life, a series of six free live sessions exploring that question. We’ll look at how people are using agents, what makes their setups work, where things go wrong, and how this changes what individuals and small teams can attempt.

I’m particularly excited about the range of perspectives. Software development is part of the story, alongside running a business, producing media, doing research and understanding the world that this technology is changing. There’s plenty here for experienced developers, and for people whose expertise lies somewhere else entirely.

Come Inside the Factory

We’re joined by great guests:

• Stefanie Hetjens (EVERYWOW), 22 September: Beyond More Output: How a Media Agency Uses AI for Quality. How EVERYWOW uses agents and builds its own tools to improve the work, with lessons from what failed along the way.

• Me, 25 September: Five Mistakes People Make with Agentic Factories. The recurring mistakes I see in mindset, setup and oversight, and how to develop a repeatable process you can trust.

• Ben Holmes (Warp), 29 September: How Warp Builds Warp using Warp Factories. A look at the product and how Warp puts it to work on its own development.

• Tony Simons (Hermes Launch Lab), 2 October: Your AI Workforce: Run an Agent Factory in Hermes Desktop. Tony’s own setup, how he runs work through Hermes, and what he has learned helping others adopt it.

• Dror Poleg, 6 October: Thinking About AI, with AI. How Dror uses AI to investigate technological change and its connections to work, the economy and society.

• Hugo, 7 October: How I Built a Media Empire With an Agentic Software Factory. How he built a factory around Vanishing Gradients, from research and production to publication and promotion.

There will be presentations, demos, live Q&A, and recordings will be available afterwards.

👉 Join Factory Life for free

The series leads into our October course, Build Your Agentic Software Factory, where you’ll set up your own factory and use it to build, verify and publish working software. You’re welcome to join the free series whether or not you plan to take the course.

Join me for Build AI Agents from First Principles

I also want to recommend Hugo’s upcoming workshop, Build AI Agents from First Principles, on 24 September. I’ll be joining it myself.

If you use agents and want to understand how they actually work, this is a great opportunity to build one from scratch. You’ll start with an LLM call, add tools and state, and build the loop that lets an agent decide what to do next. Then you’ll work through context, memory and debugging, before moving to an agent SDK and connecting tools through MCP.

The workshop uses a customer service agent as a concrete project. You should be comfortable reading and making small changes to Python, but you don’t need prior experience building agents.

I like this approach because building the pieces yourself gives you a much clearer understanding of what you’re asking an agent to do, what its environment needs to provide, and where to look when something goes wrong. Those are useful skills for anyone working with factories too.

Get a reader discount on Hugo’s workshop. The link includes the code friendsofeleanor.

Hope to see some of you you there!

When Elite AI-Assisted Coding first launched, cloud-agent delegation was early and revolutionary. Still, we insisted on giving it a respectable almost-third of the course, convinced that this is becoming the right approach.

Things have moved quickly. I now spend less time asking AI to write code and more giving agents outcomes, context, constraints, and a definition of done. They implement, check the result, deploy, manage, and report back.

This is factory mode: a persistent system of autonomous agents, with access to tools, skills, tests, and feedback loops. People retain responsibility for intent, judgement, and verification while the factory handles the implementation and administration end-to-end.

and I have explored these ideas together for some time. Hugo is an exceptional educator and adviser whose work has reached more than six million learners. He has a rare ability to make complex technical ideas clear, practical and approachable, and I’m genuinely excited to teach this course with him. I am delighted that we are teaching this together.

In our upcoming two-week course, Build Your Agentic Software Factory, you will set up our own factory, build and evolve working applications, and learn to specify, delegate, verify, and publish software. It is open to experienced developers and people who have never written code.

The cohort runs 13–22 October. The Very Early Bird price is $750 through 31 August, rising to $850 in September and $950 in October. EAAC alumni also receive an additional 20% off on top (check your email).

Join us and build your agentic software factory

It was really fun, and a genuinely interesting conversation. We talked about how my AI-assisted coding practice has changed over the last few months: less close supervision, more building systems that let agents do good work with the right context, guardrails, and verification.

We also covered async cloud-based agents, why deterministic checks matter so much, AGENTS.md and skills, test-driven workflows for agentic coding, language choices like TypeScript and Go, supply-chain concerns, and why software development increasingly feels like a branch of systems engineering.

This is the second cohort of the refreshed course, with a much stronger focus on agentic engineering: context engineering, spec-driven development, coding agents, async/background agent workflows, AI-assisted code review, PRs, testing, CI/CD, documentation, security, and enterprise-ready adoption.

The course is vendor-agnostic, so you can use Claude Code, Codex, Cursor, Copilot, OpenCode, or whatever already fits your workflow. The point is to build durable engineering practice around AI agents: better context, clearer specs, safer delegation, and higher-quality code.

Recent Maven reviews describe the course as a “masterclass in agentic engineering”, “no fluff”, and a practical bridge from theory to real workflows. Several students called out the course’s structured approach, tool-agnostic framing, and focus on state-of-the-art agentic coding.

The next cohort runs June 21–July 10, with the first week starting June 22. If you want to move beyond prompts and use AI coding agents as part of a dependable software engineering workflow, this is the cohort to join.

Join here: https://maven.com/agentic-ventures/ai-coding

In a recent session of Elite AI-assisted Coding, Eleanor hosted Greg Ceccarelli and Jake Levirne, co-founders of SpecStory and Stoa. The discussion centered on a critical challenge emerging in the age of AI-assisted development: as coding agents drastically compress the time required to write and deploy software, the primary bottleneck in the software development life cycle (SDLC) has shifted upstream.

The focus of the talk, supported by their presentation slides, was on identifying, measuring, and optimizing what happens before the first line of code is ever written.

The Shifting Bottleneck in Software Delivery

For the past decade, the software engineering industry has obsessed over optimizing the implementation and deployment phases. Frameworks like DORA (DevOps Research and Assessment) established metrics such as lead time for changes, deployment frequency, mean time to recovery, and change failure rate. However, as Greg pointed out, these metrics possess a significant blind spot: they generally start the clock at the first code commit.

“I think what’s interesting is everyone has started to play with and incorporate agentic development into their workflows that the things in the past that you might have put a lot of emphasis on like the time from something being first committed to a repository to the time that it is released to production for your customers to maybe experience have been changing.”

With the advent of coding agents and mature CI/CD pipelines, the time to implement features has collapsed from days or weeks to mere hours, and deployment times have shrunk to minutes. Consequently, the new dominant latency in the SDLC lies in the “fuzzy front end” — the period from when an opportunity or idea is identified to when a project is actually committed to and defined.

Greg introduced the concept of Lead Time to Ship, which encompasses discovery, development, and deployment. The goal is to understand how fast a decision made on a Monday translates into tangible value that a customer can experience.

Unpacking Intent Lead Time

To address the upstream bottleneck, the presentation defined a new metric: Intent Lead Time. This represents the pre-commit portion of the Lead Time to Ship, calculated as the time delta between when a decision is captured and when the first commit is made.

Greg broke down Intent Lead Time into four sub-latencies:

1. Capture: The time from a decision being made to it being formally recorded.

2. Sequencing: The time from a recorded decision to a ticket being created and prioritized.

3. Pickup: The time from a ticket existing to it being assigned to an engineer.

4. Activation: The time from assignment to the first actual commit.

In traditional workflows, these steps can take weeks. However, the introduction of AI agents fundamentally alters this topology.

“In our minds though, over time, and I don’t know if people on this call have experienced this, you know, when we’re building product internally at SpecStory, it’s almost like these three steps collapse. Like sequencing, pickup, and activation almost collapse.”

When a team can move directly from a transcribed call to an agent spinning up a prototype, the traditional ticketing and assignment phases become obsolete. The remaining friction point is the initial capture — transforming an ambiguous conversation into a structured specification that an agent can act upon.

The Challenge of Specification Validation

During the session, Eleanor raised a pivotal question regarding the verification of the intent phase. If code is increasingly just a derivative of the specification, measuring code commits becomes less interesting. The real challenge is determining when a specification is actually ready and trustworthy.

“I have no idea how to verify unambiguously that a specification is ready. That’s the thing I want to measure, right? Like when did they get to a specification that I can actually trust?”

Jake addressed this by highlighting the inherent non-determinism of product development.

“If you frame it up as like, you know, how do I know my specification is correct and complete? You don’t. Right. I think what’s happening is that this is exposing more people in the industry to the realization of the non-determinism of getting your spec right because it’s always been a problem. It’s just that we’ve found other ways to work around it.”

The solution is not to seek a perfect specification upfront, but rather to utilize agents to shorten the cycle times between market feedback and product iteration. Greg emphasized that agents allow teams to generate multiple high-fidelity prototypes rapidly, facilitating immediate validation with customers on calls, rather than relying on abstract discussions or low-fidelity wireframes.

When a student asked about the risk of customers mistaking high-fidelity prototypes for production-ready software, Jake clarified their team’s philosophy: they do not separate prototypes from production. All artifacts are treated as software delivery that is expected to be continually improved.

Leveraging Synthetic Data for Warm Starts

Aaron inquired about the potential of using synthetic data generation as a “warm indicator” to help build out specifications before they meet real users.

Jake provided a compelling example from a healthcare product consultant who uses Large Language Models to generate synthetic users based on target personas.

“He uses those synthetic users to test his spec, not to test his software. And so like he will create a set of these users that match his given personas and then he’ll start clean like clean sessions, clean context and he’ll just feed in the synthetic user and his spec and look for feedback.”

While acknowledging the challenges of ensuring synthetic personas are sufficiently distinct and salient, Jake noted that this approach provides a valuable “warm start,” potentially elevating the baseline of user feedback loops long before live customer testing occurs.

Real-Time Collaboration with Stoa

To address the logistical friction of gathering feedback and iterating on specifications, Jake and Greg introduced their new platform, Stoa. Designed for the agentic era, Stoa operates as a multiplayer collaborative environment that blends video conferencing, document editing, and shared AI coding agents into a single, durable workspace.

“So the crudest way to think about it is it’s like Google Meet meets Google Docs meets shared Claude Code agents meets instant everything’s versioned all jammed in one.”

In practice, this means a team can be on a call, discussing a feature, while an integrated agent transcribes the conversation, distills the decisions, and actively modifies the prototype or codebase in real-time. Because the environment is durable, the context — including transcripts, files, and agent history — persists after the meeting ends, eliminating the context evaporation typical of disjointed modern tooling.

Key Takeaways

The session highlighted several profound shifts in how engineering teams must operate to maintain a competitive advantage:

1. Focus on Intent, Not Just Implementation: With implementation times collapsing, competitive advantage now lives in the “fuzzy front end.” Teams must optimize how fast they can capture decisions and validate ideas.

2. The Collapse of the Ticketing Pipeline: Agentic workflows bypass traditional sequencing and assignment phases. The goal is to move directly from a captured decision to a committed prototype.

3. Specifications over Code: As code becomes a derivative product of AI agents, human authority and effort must migrate upward to focus on intent, judgment, and the authorship of robust specifications.

4. Real-Time Validation is the New Standard: The ability to build and modify software live during customer or team calls drastically reduces feedback loop latencies, rendering asynchronous prototyping cycles obsolete.

5. Durability of Context is Critical: The tooling of the future must centralize and persist the context of decisions, transcriptions, and agent actions to maintain a shared mental model across the team.

The agentic era does not eliminate the need for human engineering; it merely shifts the human focus from the mechanics of typing code to the rigor of defining value. As Greg concluded, those who engineer their processes to navigate this new paradigm will fundamentally outpace their competitors.

It is a common scenario for technical and non-technical leaders to observe that while AI programming tools drastically increase the speed at which individual developers write code, the overall software development team does not see a proportional increase in productivity.

This discrepancy occurs because writing code is only one component of a multi-participant, large-scale software development project. When individual contributors generate code faster, the subsequent stages of the development lifecycle fail to keep pace and become the new bottlenecks. These constraints typically include:

• Code review

• Testing and quality assurance

• Integration

• Release management

• Team coordination

Because these downstream processes have not been accelerated, the project as a whole can become blocked, entirely negating the speed gained during the initial coding phase. Individual AI productivity does not automatically translate to team productivity.

How to Unlock Team-Wide Productivity

To achieve true productivity gains, organizations must move beyond simply empowering individuals to write code faster. The solution requires rethinking the entire software development lifecycle and infusing AI into every phase.

1. Scale AI Beyond Code Generation: Integrate AI tools into your code review process to quickly unblock developers. Utilize AI in your testing frameworks to automatically and thoroughly validate code, reducing the reliance on manual, individual testing.

2. Leverage Continuous AI and Autonomous Agents: Implement agentic workflows to handle the routine “glue work” of integration and deployment (CI/CD). These continuous AI systems can operate autonomously in the background and report back when tasks are complete, freeing up developers for higher-level problem-solving.

3. Cultivate Psychological Safety: The human factor is just as critical as the technology. Ensure that your team operates in an environment where they feel safe to regularly experiment with AI tools. Building a habit of reflecting on how AI is integrated into the workflow — and sharing those experiences across the team — allows the group to iterate and move faster.

By systematically addressing these bottlenecks and fostering a culture of shared experimentation, teams can achieve substantial productivity improvements alongside enhancements to quality and innovation.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

Test-Driven Development (TDD) is exceptionally powerful when combined with AI agents. While historically considered time-consuming for human developers, TDD provides a mechanistic verification process that is perfectly suited for AI. By relying on tests, you eliminate the risk of the agent generating inadequate code or solving the problem incorrectly. Furthermore, it removes the need to manually verify — or “babysit” — every step of the agent’s work.

Agents possess a unique strength in rapid iteration and error resolution. When presented with a set of predefined tests, an AI can repeatedly adjust its implementation until the solution is correct, transforming TDD into a highly reliable and low-overhead methodology.

The Agentic TDD Workflow

To effectively guide an agent through TDD, implement a strict orchestration pattern:

1. Write the Tests First: Begin by developing a robust test suite. Ensure you have comprehensive coverage that addresses all critical and interesting aspects of the desired software. This step locks in the requirements before the agent writes a single line of implementation code.

2. Isolate the Agent and Protect the Tests: Delegate the implementation phase to the agent, but enforce strict boundaries. The agent must be completely isolated from modifying the tests. In some configurations, it is beneficial to prevent the agent from even viewing the test source code, restricting its visibility solely to the test execution results and error logs.

3. Iterate Until Passing: Allow the agent to iteratively write and refine the implementation. Because the agent excels at iterating against predefined constraints, it will continue adjusting the codebase until the entire test suite passes.

Automation vs. Manual Orchestration

Depending on your tooling and confidence level, this process can be managed in two distinct ways:

• Fully Automated: With proper prompting and orchestration frameworks, the entire loop of generating code, running tests, and feeding errors back to the agent can be delegated entirely. The agent independently works with discipline through the feedback loop to reach a passing state.

• Manually Orchestrated: For more complex problems, you might prefer to oversee the cycle. In this approach, you pay close attention to the failing tests, provide minor guidance as the agent iterates, and independently verify that the final implementation passes without any unauthorized alterations to the test suite.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

Yes, in terms of sheer reliability, but relying exclusively on context files sacrifices scalability. This architectural decision should be viewed as a trade-off rather than a strict binary choice.

Recent research from Vercel confirms a common intuition among AI practitioners: context loaded upfront in a file like AGENTS.md or CLAUDE.md operates consistently more reliably than the exact same context loaded on demand through Agent Skills.

However, this does not mean you should avoid using Agent Skills. Instead, you must balance the model’s need for constant awareness against the system’s need for an expansive, modular toolset.

Understanding the Trade-Off

To build an effective agentic system, it is important to understand the mechanical strengths of both approaches:

• Upfront Context Files: Files like AGENTS.md are loaded immediately and remain constantly visible to the agent. Because the model processes this text continuously, it is highly reliable. You should use upfront context when the agent requires strict, persistent adherence to core instructions or when the rules must always be top-of-mind.

• Agent Skills: Skills are loaded progressively on demand. The agent only reads the brief descriptions of the skills upfront, fetching the underlying instructions and logic only when a specific skill is invoked. This allows you to maintain a vast library – sometimes hundreds of distinct capabilities – without flooding the context window or degrading reasoning performance. Skills are ideal for explicit tasks where the trigger for using them is obvious.

The Hybrid Solution

In practice, the most robust architectures combine the two mechanisms. You can leverage the persistent reliability of upfront context to manage the dynamic scalability of Agent Skills.

To implement this hybrid approach effectively, divide your instructions as follows:

1. Store bulk content in Agent Skills: Place the heavy, verbose elements — such as full scripts, detailed reference materials, extensive assets, and complex execution steps — inside individual Agent Skills.

2. Place triggers in the upfront context: Use your AGENTS.md or CLAUDE.md file to provide precise operational hints. Tell the agent exactly when to load specific skills and how to apply them to a given situation.

By using the upfront context as a reliable index and routing system, the agent will consistently know which tools to reach for, while the Agent Skills will ensure your overall system remains highly scalable and your context window remains uncluttered.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

…

You do not need a deep technical background in machine learning to create powerful AI systems. Whether you are a software engineer, a business manager, or a creative professional, anyone can become an effective AI operator today. The most practical approach is to start building directly, focusing on orchestration and evaluation rather than the underlying mathematical theories.

Here is a recommended, step-by-step progression for getting started:

1. Master “Harness Engineering” with Local Agents

If you want to build complex systems, the best entry point is working interactively with local coding agents. Excellent options include Claude Code, Copilot CLI, Codex, and OpenCode. These tools offer a rich vocabulary of customization mechanisms that allow you to orchestrate how the AI operates.

You should focus on mastering features such as:

• Agent skills and custom instructions.

• Subagents for delegating specific, modular tasks.

• Hooks that connect the AI to external tools, scripts, or workflows.

Using these orchestration mechanisms — an approach now referred to as harness engineering — allows you to build custom automations, complex workflows, or even complete applications tailored to your specific needs.

2. Transition to Programmatic Execution

Once you have successfully built a workflow that functions well interactively, the next phase is to drive it programmatically. All of these major coding agents provide Software Development Kits (SDKs), such as the Claude Agent SDK, OpenCode SDK, Codex SDK, or Copilot SDK.

By utilizing a local agent SDK, you can shift from manual interaction to full automation. This enables you to:

• Run your AI workflows as scheduled background jobs.

• Trigger AI processes automatically in response to specific system events.

• Seamlessly integrate the AI capabilities into larger, pre-existing applications.

3. Adopt an Empiricist Perspective

Working with AI is fundamentally different from writing traditional code. AI models are non-deterministic, meaning the same input might yield varying outputs. To build dependable systems, you must learn to measure and manage this inconsistency.

As you continue finding new approaches and tricks, you should focus on:

• Data science and statistics: Learn the basics of how to measure and interpret varied outputs across large datasets.

• Evals (Evaluations): Implement systematic testing frameworks to score the quality, accuracy, and reliability of the model’s responses.

• Experimentation: Run structured tests to observe exactly how modifying your harness (like tweaking instructions or adjusting hooks) affects the AI’s behavior in production.

You can begin building with AI immediately. Focus entirely on using a local coding model’s full feature set to customize your workflow, transition to an SDK for automation, and rely on robust data science and evals to guarantee dependable results.

While there is considerable excitement surrounding multi-agent orchestration frameworks, you should generally treat these open-source packages as experimental research projects rather than production-ready tools.

It is highly recommended to explore them for educational purposes, but integrating them into active workflows or commercial software is usually premature.

The Limitations of Scaffolding

A common maxim in artificial intelligence development is that capabilities not yet mature in the underlying models cannot easily be forced into existence through complex scaffolding or harnesses.

Attempting to compensate for a base model’s current limitations by tacking on heavy orchestration infrastructure typically leads to negative outcomes:

• Decreased Efficiency: Adding layers of agent management and communication often slows down execution and complicates debugging.

• Increased Costs: Running multiple agents in a complex, overlapping harness drives up API usage and compute expenses without a guaranteed return on output quality.

How to Utilize Open-Source Multi-Agent Orchestration Frameworks for AI Coding Today

Instead of deploying these systems in production, use them as learning tools. You can benefit from these public experiments in the following ways:

• Study the Architecture: Download the code and read through it to understand how different developers approach the complex task of scheduling and orchestrating agents working in concert.

• Monitor Developments: Keep a close eye on these projects — especially when a new generation of base models is released. Framework authors often rush to exploit new native capabilities, which can provide valuable insights into the shifting state of the art.

The Recommended Alternative

For current complex, long-running software projects, you do not need a third-party multi-agent framework. Instead, rely on the native capabilities of established coding harnesses like Copilot, Codex, or Claude.

These platforms typically provide everything you need through a more straightforward, reliable structure:

• Main Agent Thread: Acts as the central harness to manage the overarching task, drive the primary logic, and maintain long-term context.

• Sub-Agents: Support the main thread by specializing in specific sub-tasks, farming out discrete workloads, and managing localized context.

Future Orchestration Needs

When the time eventually comes that your workflow genuinely requires a robust multi-agent orchestration setup, you will likely be better served by building a custom system. By that point, you will have a clear understanding of your precise needs and can design a bespoke architecture that addresses them directly, rather than wrestling with a generalized, experimental open-source package.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

…

It is common to set instructions – whether in an AGENTS.md file, a CLAUDE.md file, or another rules file – only to find the agent ignoring them or applying them inconsistently.

To fix this, you need to understand why it happens: instructions are merely text appended to the model’s context for every request. Because large language models are inherently non-deterministic, they may occasionally misunderstand, de-prioritize, or entirely miss your directives.

The most effective approach is to treat your agent instructions as project infrastructure that requires continuous testing, verification, and evolution.

Strategies for Debugging and Refining Instructions

Here are several practical methods to ensure your agent consistently follows your guidelines:

• Test and Verify Repeatedly: Do not assume an instruction works based on a single successful output. Force the agent to invoke the specified behavior multiple times. For instance, if you mandated a specific commit message style, trigger several commits to verify that the format is strictly and consistently applied across different scenarios.

• Interrogate the Agent: Open a direct chat with the agent and ask it to explain its understanding of your instructions. Present hypothetical scenarios by asking, “What if I ask for this specific task to be carried out? How would you do it? Give me a clear and detailed step-by-step explanation.” This allows you to audit the agent’s interpretation before it performs actual work.

• Inspect Thinking Traces: If your model emits reasoning traces, review them carefully. These “thoughts” provide valuable clues about which parts of the context the model is paying attention to and whether it actually processed your specific instructions before generating its final output.

• Iterate Like a Prompt Engineer: Treat your configuration file as you would any AI prompt. If an instruction is ignored, try rewording it, moving it higher in the document, or repeating it. You can even use ALL CAPS to draw attention to critical, non-negotiable behaviors.

• Ask the Agent for Help: If you are struggling to enforce a rule, ask the agent itself how best to phrase the instruction. It can often help you write a more effective prompt tailored to its own reasoning patterns.

The Importance of Consistency

Always remember that models are non-deterministic – the exact same instructions in the exact same environment will yield slightly different results on each run. You must run multiple trials of the same scenario to guarantee consistency. Iterate and refine your instructions until the desired behavior is fully understood and reliably executed.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

With new AI models being released almost daily, featuring continuously improved capabilities and performance, selecting the best one for your specific needs can be challenging. To effectively evaluate and choose the right model, you should rely on a combination of independent benchmarks, crowdsourced testing, and personal validation.

Here is a structured approach to evaluating and selecting AI models:

1. Artificial Analysis

For a high-level overview of currently available models, Artificial Analysis is an excellent starting point. They conduct their own independent testing rather than relying on the metrics published by model providers, which lends their results significant credibility. This resource is particularly useful for evaluating the high-level trade-offs between:

• Performance: How well the model executes standard benchmarks and general tasks.

• Cost: The price per task, token, or API call.

• Speed: The latency and throughput of the model during inference.

2. Arena

Formerly known as LM Arena, Arena takes a dynamic approach to model evaluation. Instead of using a fixed set of benchmarks, it hosts a competitive league where models are pitted against each other.

• Crowdsourced Blind Battles: Users conduct blind tests by prompting different models with the exact same task and voting on the best response.

• Real-World Coverage: Because users have infinite imagination for new applications, this method captures a much wider array of use cases than static benchmarks — a crucial advantage for evaluating models.

Often, your own intuition about a model’s strengths and its optimal use cases will align closely with the crowdsourced consensus found on Arena.

3. Personal Testing

Most importantly, public testing platforms and data can only take you so far. To truly determine which model works best for your specific application, you must conduct your own testing. You should approach this phase strategically:

• Run Familiar Tasks: Maintain a baseline set of tasks that you have already processed with several older models. Because you know what to expect from these prompts, running them with a new model will give you a precise understanding of how its behavior compares to its predecessors.

• Retry Past Attempts: Always revisit complex tasks that previous models struggled to complete. The enhanced reasoning capabilities and improved performance of the latest generation of models frequently unlock solutions that were previously impossible to achieve.

General Guidance

When researching a new model for your agentic tasks, start by checking Artificial Analysis for empirical trade-offs, consult Arena for crowdsourced performance in real-world scenarios, and finally – and most importantly – conduct rigorous personal testing to validate the model against your specific requirements. Public data informs your intuition, but personal testing verifies the reality of the model’s capabilities in your production environment.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups. Join us!

While agentic engineering is incredibly powerful and shows great promise, its capabilities are sometimes overhyped. Reports of AI agents ‘one-shotting’ complete applications from scratch usually involve simple projects where the agent cannot really go wrong. In those scenarios, as long as the generated code compiles and does something useful, it is considered a success.

Applying agentic coding to more complex, existing projects with many interdependent components and legacy code requires a deliberate approach. To achieve high-quality, dependable results, two foundational practices are crucial: Context Engineering and Specification.

Context Engineering

Context engineering involves providing the agent with project-specific instructions and background knowledge. This grounds the agent in the reality of your codebase, ensuring its outputs align with established patterns. Key methods include:

• Configuration Files: Using dedicated files, such as AGENTS.md, to outline base rules.

• Progressive Loading: Supplying context dynamically through specific Agent Skills.

• Project Documentation: Sharing architectural decisions, coding conventions, and system designs.

• Workflow & Testing Guidelines: Providing explicit information on how to use the project’s infrastructure and test suite.

Specification

Specification is the practice of guiding the agent step-by-step using complete, unambiguous information about the immediate task. Rather than relying on the agent to infer your goals, you must define the boundaries explicitly. This can be achieved through:

• Spec-Driven Development: Utilising comprehensive specification documents, such as formal product requirements or detailed roadmap instructions.

• Detailed Prompting: Crafting prompts that clearly articulate the core intent, constraints (what the agent should not do), what a successful outcome looks like, and how to verify that success.

By rigorously applying both context engineering and specification, developers can tame the inherent unpredictability of agentic engineering – transforming it into a dependable approach even within the most intricate legacy systems.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups, and we’re offerring a 25% early bird discount until March 23rd. Join us!

Yes, but in almost all cases, using the best cloud-hosted models is the recommended approach.

Many teams hope to use locally-hosted open models for agentic coding, typically driven by two main motivations: reducing the cost per token and meeting strict privacy or security compliance rules. While open models are impressive and rapidly improving, hosting your own models locally is rarely the most practical or effective solution.

There are three primary factors to consider when making this decision:

1. Model Capabilities

The most advanced open models are still a few steps behind the frontier models developed by the main research labs. When building and deploying agentic projects, you generally want access to the absolute best capabilities available. Cloud-hosted frontier models currently provide a significant edge in reasoning and coding proficiency.

2. The True Cost of Hosting

At first glance, locally-hosted models look great on paper because the raw cost per token is much lower. However, this calculation often ignores the full cost of ownership. When you factor in the hidden expenses, the financial reality looks much less favorable. These hidden costs include:

• Procuring high-end hardware.

• Installing, configuring, and maintaining the infrastructure.

• Amortizing hardware costs over several years.

• Dealing with low utilization rates compared to the highly optimized infrastructure of a cloud provider.

3. Privacy, Security, and Compliance

Operating in environments under strict compliance rules is the primary scenario where a locally-hosted model might be an absolute necessity. If a cloud provider is strictly prohibited by policy, local hosting is a vital option to have.

In reality, however, most compliance needs can be met by cloud providers. When examining actual regulatory requirements, the commitments from cloud labs — including end-to-end encryption and strict data privacy guarantees — are typically more than sufficient for most organizations.

Making the Final Decision

When evaluating locally-hosted open models versus cloud-hosted frontier models, the trade-offs are usually clear. Open models offer an environment where you maintain absolute control, which is necessary for environments with the strictest compliance rules. They also offer a low raw cost per token. However, this is offset by high hidden costs, including hardware procurement, ongoing maintenance, and the inherent inefficiencies of low utilization.

Conversely, cloud-hosted frontier models deliver state-of-the-art performance that is best suited for complex agentic coding tasks. They are generally more cost-effective overall because they avoid expensive hardware procurement and maintenance. Furthermore, their high security standards, paired with end-to-end encryption, satisfy the vast majority of compliance needs.

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups, and we’re offerring a 25% early bird discount until March 23rd. Join us!

An outtake from Sunday School: Drop In, Vibe On.

In this session, Eleanor Berger hosted Zain Hoda to explore X1PM, a lightweight, shared workspace designed to bridge the gap between human collaboration and AI agents. While current agentic workflows often rely on complex tool calling for software like Google Workspace or GitHub, X1PM proposes a radically simplified approach: interacting purely through standard Markdown and CSV files.

This presentation showcased how shifting away from heavy platform-specific formats to an agent-native file system allows both humans and multiple AI agents to read, write, and collaborate on the same set of documents seamlessly.

The Motivation Behind X1PM

The core philosophy of X1PM stems from the observation that while AI agents are generally excellent at navigating traditional codebases and file systems, they often struggle when presented with overly complex tools.

“A lot of people have had the observation that agents are really good at interacting with file systems. And so I think a lot of what people have discovered is that you know if you connect your agent to MCPs that are extremely complex they kind of struggle with that.”

Zain noted that connecting agents to dense services, such as a Google Workspace MCP or a GitHub MCP with dozens of available tools, can lead to context rot and confusion for the models. The solution was to return to basics, leaning into file formats that agents process naturally. In Zain’s personal workflow, he effectively replaced Google Docs with standard Markdown files and Google Sheets with simple CSV files.

Although this approach works locally, it becomes challenging when trying to access those documents from the web or from a mobile client. X1PM solves this by providing an interface that looks and feels like a structured workspace but operates on top of these raw, agent-friendly files. It acts as an orchestrator, or a shared space, exposing standard file-system operations — such as reading, writing, searching, and editing — via an MCP server.

Building a Unified Agent Experience

Eleanor noted that this approach beautifully addresses a pain point many AI-native developers experience when using local filesystem-based tools and workflows.

“I love this approach of using markdown files and CSV files because you don’t need all this heavy software... but what I realize that I’m missing is a space that makes it easier for me to collaborate with the agent.”

With a local workflow, connecting an agent on a phone or jumping to a web UI is difficult. X1PM bridges the gap between the structured, readable UI of a SaaS application and the simple file-system access required by coding agents.

The Live Demo

During the session, Zain provided a live demonstration of X1PM in action, emphasizing how easily an agent can interact with the structured data. He used a lightweight CRM system built entirely from a CSV file.

He initiated an interaction using the Claude app, instructing the agent to update a specific contact’s status in the CRM.

“I’m going to tell Claude in the demo CRM, change Emma Wilson’s status to closed won. And this is like full live demo, so I have no idea if this is going to work.”

The agent quickly listed the files, searched the structure, and correctly updated the CSV file. A quick refresh of the X1PM web interface confirmed the change, demonstrating the zero-friction collaboration between human intention, agent execution, and the resulting shared truth in the file.

Cross-Agent Continuity

One of the most striking aspects of the workspace is the ability to run multiple tools seamlessly over the same context. Zain demonstrated switching between Claude, a custom coding agent in a GitHub Copilot setup, and the Cursor IDE.

He tasked Claude with generating the requirements and file structure for a “GTM Engineering” course. Claude set up a directory populated with a README.md, tasks, curriculum, and a launch plan.

Because X1PM is exposed as a file system, Zain could then switch over to Cursor and ask it to begin building out a landing page for the project based on the context Claude had just generated. Cursor had instant access to the newly created folders and files.

Eleanor pointed out the significant benefit of having this level of consistency:

“It’s also really cool that you have this consistency because the interface is so simple and this file system tools actually can switch agents, switch models, and they’ll all kind of well assuming the models are good enough, they all kind of still figure out what to do.”

By rooting the context in standardized text files, the environment becomes completely agnostic to the specific model, UI, or IDE being used.

Going Beyond Coding

X1PM also highlights the potential for agents outside of sheer code generation. Because the system utilizes standard CSVs, developers can orchestrate background agents to perform scheduled logic over the data.

Zain offered the example of a freelance developer managing leads out of a CSV. An autonomous agent could be scheduled to regularly scan the file to flag clients who require follow-ups. A secondary agent could ingest this flag and draft personalized emails based on a Markdown template found in the same workspace.

“You have a lot of visibility into all of the context that your agent has. So really the way to think about this is like this is the sort of shared context space between you and your agents.”

This concept resonates strongly with those running complex multi-agent architectures or needing a centralized place to monitor non-code outputs, such as content calendars or style guides.

Managing Conflicts and Coordination

A naturally arising question when discussing multi-agent systems sharing a file system is the risk of conflicting edits or data loss. A viewer asked if there were safeguards against multiple agents overwriting each other.

Zain shared that, currently, X1PM handles this by returning a standard file-locking error if an agent attempts to write to a file currently being modified by another. The agent will typically catch the error and retry.

He also expanded on how these rudimentary text files could act as a sophisticated coordination layer. Referencing recent experiments by Anthropic where large swarms of agents were deployed to rebuild a C compiler, Zain noted that they relied on lock files to prevent conflicts.

“Those locks could essentially just live in a CSV file and you could tell the agent that when you need if you intend to write... first update the CSV file in X1PM and then proceed with it.”

As developers deploy heavier multi-agent workflows, a central CSV indicating which files are currently “checked out” by which agent becomes an invaluable, human-readable coordination tool.

The Future of Shared Agent Workspaces

The discussion then pivoted to the broader implications for the development ecosystem. Another participant shared their experience building custom systems using Markdown files for task prioritization and tracking among multiple agents. The participant noted that X1PM’s cloud-hosted nature offered an immediate benefit by providing a clean, accessible interface to monitor the progress of these ongoing autonomous tasks without having to constantly comb through local Git repositories.

Eleanor noted that other platforms are beginning to explore this space as well, mentioning Thomas Dohmke’s upcoming “Entire” platform which similarly aims to tackle human-agent collaboration. However, tools natively built from the ground up to support this file-system-first architecture remain scarce.

Zain was pragmatic about the future of his tool, noting it started as a solution to his own daily problems.

“Honestly like it would like something like this will eventually become like a feature of the some other product or other sets of products or maybe some new product comes you know with more polish around this. But for now, like I use it daily.”

Whether X1PM grows into a massive independent product or its underlying philosophy is absorbed by larger platforms, the core execution — simplifying the interface into accessible, text-based file systems — acts as a blueprint for the future of agent tooling.

As AI agents continue to scale, building systems that are natively understandable by both humans and machines will be critical for productive, observable workflows.

…

Looking to learn how to use AI agents effectively for software development? The spring cohort of Elite AI-Assisted Coding, the #1 comprehensive course on making the most of agentic software development, is open for signups, and we’re offerring a 25% early bird discount until March 23rd. Join us!

Hugo is one of those rare educators who actually builds production systems and knows how to teach. That combination is hard to find.

Hugo and Stefan Krawczyk are teaching the next cohort of Building AI Applications for Data Scientists and Software Engineers, starting March 10th (I’ll be there too – this time as a student). The course covers the full lifecycle of shipping LLM-powered applications — not toy demos and POCs, but real production systems with proper evaluation, debugging, observability, and deployment.

Here’s what caught my attention:

• 4 weeks of live, hands-on training with Hugo and Stefan – deep experts and gifted teachers

• Guest speakers from DeepMind, Pydantic, Chroma, and more

• Over $1,000 in credits from Modal, Pydantic Logfire, Baseten, Braintrust, Chroma, and others

• 250+ satisfied alumni from Netflix, Meta, TikTok, Adobe, Amazon, Ford, and others

Most LLM courses teach you how to call an API and write a prompt. This one teaches you how to build something durable and dependable.

Hugo and Stefan are offering a 🔥 25% discount to members of this community. If you’ve been wanting to level up from prototypes to production AI systems, this is the one.

See you in class.

An outtake from Sunday School: Drop In, Vibe On.

In this session, Eleanor Berger explored a different facet of agentic AI. While previous sessions often focused on pure coding tasks — building backends, CLIs, or agents themselves — this talk centered on using agents to facilitate creative work: specifically, the creation of presentation slides.

The motivation came from a practical need. Eleanor had been preparing materials for the Elite AI Assisted Coding course and found herself dissatisfied with her initial slide designs.

“I was not so happy with the quality of slides I had for a first instance... I realized this is an opportunity to do something better.”

The result was sunday-slides, an agent skill that manages the process of presentation creation — from content generation and layout to illustration and export. This workflow demonstrates how interactions with AI can move beyond simple question-and-answer exchanges to multi-step processes involving various tools and sub-agents.

The Architecture of the Skill

The sunday-slides skill includes several components:

• SKILL.md: The core instruction file describing the workflow.

• Assets: A bundled logo (customizable) and styling resources.

• Scripts: Python code for specialized image processing.

• Configuration: CSS and TypeScript files for the slide framework.

Interactive Configuration

One of the first things the skill does is establish the constraints for the specific presentation being generated. The skill uses the “Ask User Questions” tool to ask the user about their preferences.

“It wants to use the user question tool. That’s the thing that shows you a UI for answering different multiple choice questions for these parameters and that’s very convenient.”

This step determines several visual parameters:

• Background Color: Setting the tone for the deck.

• Image Style: Determining whether illustrations should be photorealistic, vector art, or — as chosen in the demo — retro pixel art.

• Logo: confirming whether to use the default branding or a custom asset.

These responses are stored as variables to maintain consistency throughout the generation process.

Leveraging Sli.dev

Rather than trying to reinvent presentation software, the skill leverages Sli.dev (Slidev), a powerful tool for developers that generates slides from Markdown.

This demonstrates “skill composition.” She didn’t write the Slidev integration from scratch. Instead, she installed an existing Slidev skill, reviewed it for safety, and then built her new workflow on top of it.

“This in itself comes with a skill. So I installed their skill which is really very high quality... it helped me agentically create my own skill relying on it.”

The sunday-slides workflow instructs the agent to:

1. Initialize a new Slidev project.

2. Install necessary dependencies.

3. Configure themes and styles using bundled CSS/TS files.

4. Generate the content in Slidev’s specific Markdown format.

This approach allows the agent to focus on content and orchestration, while using a specialized tool for rendering and formatting.

Parallelism with Sub-Agents

The workflow generates images in parallel. A presentation might need 10, 20, or 30 distinct illustrations. Generating these sequentially would take significant time.

Eleanor’s solution leverages the sub-agent capability found in modern AI tools like Claude Code, OpenCode, and GitHub Copilot.

“If I can do all of these images in parallel there’s nothing stopping me. Then they don’t have to be serialized.”

For each slide requiring an illustration, the main agent spawns a sub-agent. These sub-agents run concurrently, reducing the total time required to build the deck. This pattern — fanning out work to independent workers — is a standard pattern in agentic workflows.

The “Reflective” Prompting Pattern

Inside each of these sub-agents, the instruction set for generating the image prompt follows a “reflection” pattern:

1. Deliberate: Read the slide content and identify core concepts.

2. Brainstorm: Come up with several distinct visual approaches.

3. Reflect & Select: Evaluate the options and pick the best one.

4. Construct: Finally, write the prompt for the image generator.

“This is a pattern that you’ll see a lot in agentic workflows and often produces better results... it means more iterations, more tokens, that’s the price you pay for getting a better result.”

By forcing the model to “think before it acts,” the quality of the resulting visual metaphors is often higher than a zero-shot attempt.

Solving the Background Problem

A common challenge with AI image generation is consistency, particularly with backgrounds. Even when given specific color codes, models like Nano Banana often produce slight variations that look messy when placed on a slide deck.

“Even if you give them the exact color code... every time you get something a bit different.”

Eleanor’s workaround was simple. Instead of trying to match the slide’s background color, she instructed the agent to:

1. Generate all images on a pure white background.

2. Use a specialized Python script to programmatically replace the white pixels with transparency.

3. Overlay the transparent image onto the slide’s CSS background.

The skill includes this Python script — written by an agent — which uses the pillow library to handle the processing. This ensures that every illustration sits perfectly on the slide, regardless of the background color chosen by the user.

The Live Demo

Eleanor ran the skill live against a fresh topic: “Claude Opus 4.6 vs GPT-5.3-codex”.

She initiated the workflow, and the agent presented the configuration questions. Eleanor selected a Retro Pixel Art style.

The terminal output showed the orchestration unfolding:

1. Project Setup: The Slidev directory structure appeared.

2. Content Generation: Markdown files for the slides were written.

3. Image Fan-Out: Multiple progress bars appeared as sub-agents launched in parallel to generate pixel art for each slide.

“It’s running in parallel which saves some time. This is what I really like about sub agents... being able to fan out a parallel task to multiple agents is something I find very very useful.”

Within moments, the process was complete. The agent ran the build command, and a PDF was exported.

The result was a coherent presentation. The pixel art illustrations matched the retro theme, and the background removal script allowed them to integrate with the slide design.

Key Takeaways

The sunday-slides project illustrates several concepts for AI-native engineers:

• Iteration is Key: Eleanor didn’t write this complex skill in one pass. It took about 2-3 hours of iterative work with an agent — trying instructions, seeing where they failed, adding constraints, and refining the workflow.

• Composition Over Creation: The skill was built on other existing skills. The Nano Banana Pro skill for image generation and the Slidev skill for presentation building. Integrating existing tools is far more efficient than building from scratch.

• Parallelism: For batch tasks like asset generation, the ability to spawn sub-agents helps improve performance.

• Asset Processing: Sometimes the best way to solve a “prompting” problem (like background consistency) is with a deterministic script. Agents can write and run code to fix the output of other models.

“To me that’s a wonderful thing that I can do it and I think it’s in a nice demonstration of how you can use skills to build these complex workflows.”

Mastering Agent Skills is a hands-on workshop for AI power users. Join the next session this Sunday to learn how to build your own skills.