AI Engineer Code Summit 2026 Day 2 Afternoon: Automating with Calibrated AI and Running Agents Reliably

The afternoon block on Day 2 of the AI Engineer Code Summit 2026 at the Hilton Union Square in San Francisco gave me two talks that approach agents from different directions. Diogo Almeida, co-founder and CEO of TypeSafe AI, asked why AI is so useful for assisting people and so unreliable for automating work. Fabian Hedin, CTO of Lovable, described the runtime, the computer, and the verification loops that Lovable builds around its agents. These are my notes from both talks, along with photos of Diogo’s slides and what I think the ideas mean for teams in the Philippines.

If you have not read the earlier posts in this series, my notes on the Day 2 morning keynotes and the Day 1 workshops cover the layers of the agentic coding stack that these two talks build on.

Why AI assists well and automates poorly

Diogo’s session was titled I love you so jevving much, and his slides refer to TypeSafe’s model as “Jev.” He opened with a meme that asked what is going on in AI and where all the automation is. His answer was a short list, which I have reproduced from the slide below.

The slide above is the part of the talk I found most helpful. It sorts tasks into two columns. The left column, labelled too good to be true, lists instruction following, ChatGPT, copilots, coding agents such as Claude Code, NotebookLM, DeepResearch, voice agents, customer service without decisions, and slop and spam. The right column, labelled too bad to be useful, lists customer service, drive-thrus, accounting and taxes, QA, insurance, data entry, and anything that requires reliability. A red arrow points at the dividing line and asks what explains it.

Customer service appears on both sides of the line, and I think that detail is the key to the slide. Answering questions with a person ready to correct the answer falls on the left. Making the decision itself falls on the right. Diogo’s explanation is that RLHF is part of the problem. A model trained to please a human reader is shaped around a human in the loop, and a later slide made the point with a meme labelled optimization, humans, and the loop. The conclusion on his slide was short: you get what you optimize for.

A bitter lesson with two more layers

Diogo built the next part of the talk on Richard Sutton’s essay The Bitter Lesson, which argues that general methods that make use of more computation tend to beat approaches built on human knowledge. His first pyramid showed two layers, algorithms resting on compute. The next slide, titled A Bitterer Lesson, added data underneath. The final slide, titled The Bitterest Lesson, added a new foundation below everything else, labelled “doing the right task.”

I read this as a claim that compute, data, and clever algorithms only pay off when they are aimed at the task you actually want done. For Diogo, that task is automation. A later slide drew a consequence from the same idea. Because you get what you optimize for, the slide said, benchmarks do not matter, which I understood to mean that a benchmark score says little about whether a model can run a process unattended. The slide was titled “Jev’s secret” and named reliable intelligence as the reason people loved the model over copycats. It explicitly ruled out speed, cost, and interface as the explanation.

How calibrated confidence lets software automate decisions

The technical centerpiece of the talk was RLCD, which stands for reinforcement learning for calibrated decisions. Diogo developed the technique through TypeSafe, and Jev is the model TypeSafe built with it. RLHF trains a model on human preference, so raters rank outputs and the model learns to produce the ones people like. The training signal for RLCD is whether a decision turns out to be correct, and the model learns to return that decision together with a confidence score that matches how often such decisions are correct.

This treats the output of an AI model like the output of a classic machine learning classifier. When the model reports 80 percent confidence, the event should happen about 80 percent of the time. That property is called calibration, and it is what turns a model’s answer into something that ordinary software can use to make a decision safely. According to MindStudio’s write-up of the approach, Jev returns a typed decision and a calibrated confidence number instead of conversational text, so software can act automatically above a chosen threshold and send everything below it to a person. The same write-up notes that TypeSafe’s speed, cost, and error-rate figures are self-reported and unverified by third parties, and that Jev is not yet publicly available.

A concrete example shows why this matters. Suppose a workflow only accepts an automated decision when the score is above 95 percent and sends everything else to a person. That rule works only if a 95 percent score really means that about 19 out of 20 such decisions are correct. Assistant-style models do not usually give us a number we can treat this way, so teams fall back to keeping a human on every step. I noticed that this echoes the morning talk from Cloudflare, where Michelle Chen described small models that return a confidence score with every answer.

Treating models as building blocks of software

Diogo spent the second half of the talk on how this changes the way we write software. One slide placed the classic logic gates AND, OR, and NOT next to three model-backed blocks labelled NOUL, SCORE, and CHOICE. The first takes two inputs, labelled D and Q, and returns the probability that the answer is true or false. The second returns a score, and the third returns a probability for each option in a list. The slide called these “infinitely replicable lego blocks” for software.

A companion slide, titled Theme: real software engineering, listed what building with these blocks involves. It carried the caption “This is not your vibecoder’s AI.”

• Break the task down into smaller pieces (decomposition).
• Think about stages that can be verified (encapsulation).
• Design what a smart person would need in order to do each stage (state management).
• Engineer around the task itself.

He also raised the principal-agent problem, which in this setting means that we find it hard to express what we really want in software. I took the decomposition steps above as his practical answer. Each small stage has a narrow job that a person can specify and check, so the gap between what we ask for and what we want has fewer places to hide.

An AI type checker for meaning

The slide on semantic dependent typing is the clearest example of the idea. A conventional type system checks structure. If three functions named redact, summarize, and report all take a string and return a string, the shapes match and the compiler is satisfied. Diogo proposed that an AI type checker could verify meaning as well. If redact returns text marked as containing no personal information, and summarize preserves that property, then report can be shown to return text with no personal information.

This example resonates with me because many Philippine teams handle personal data under the Data Privacy Act of 2012. A pipeline that redacts customer details before summarizing a call transcript often depends on someone remembering that the redaction must hold at every later step. A check that follows the guarantee through the whole chain gives a team something much closer to evidence.

Agent design is shaped by the KV cache

Another slide, titled Agents beyond the KV cache, asked why so much of agent design is dictated by the KV cache. It posed three questions: why memory grows linearly with the conversation instead of logarithmically, why state is not shared between agents, and why routing and subagents that are more sophisticated than the naive versions cannot work. The slide pointed to Diogo’s own post, (KV) Cache Rules Everything Around Me.

The two pie charts show the main finding. Developers tend to assume that generating output is the largest cost of running an agent, and the chart on the left puts output at roughly three quarters of the bill. The chart on the right, which shows what agents actually cost, puts cache reads at roughly two thirds. The post’s summary states that the biggest cost to developers of an agent is re-reading its own context, which costs the provider close to nothing to serve. A footnote on the slide says that the actual costs assume a 100 percent cache hit rate, so the real split depends on how well a given agent keeps its cache warm.

Automate everything

Diogo closed with a call to action. If you are a software engineer, you think in the world of bits, and he asked what would happen if you were no longer constrained by them. His slide for that idea asked what we could not automate, and answered with a meme that said to make all the software smart.

Earlier slides gave his answer to the question of what Jev is for the world. It is AI made for automation, programmable, built to run in the background, and made for developers. His slide turned the line “software is so over” into “we’re so back,” the fear of a software apocalypse into a software boom, and the idea of engineers as a permanent underclass into a golden age for software engineers with early-internet energy. A slide titled “AIE secret” added that people in the field still have “LLM Stockholm syndrome” and that many ideas replace one LLM with another instead of rethinking software.

He ended by thanking the early believers, reliability realists, software researchers, dreamers, and contrarians in the room.

Lovable’s runtime for agents

Fabian Hedin’s talk, What it takes to set intelligence free, focused on the infrastructure around a model. He organized his material into three parts: the agent runtime, a computer for agents, and verification. I do not have slide photos from this talk, so what follows comes from my notes.

The agent runtime keeps its task history outside the process

The center of the runtime is an agent control plane, or ACP. It starts agents and wakes up other agents, which makes it essentially an orchestrator. Everything it coordinates is recorded in a task history, a log of events such as a request, a tool call, a result, and a question like “What’s the status?” Four properties of that design stood out to me:

I noticed a connection to Diogo’s talk that neither speaker mentioned. An append-only history keeps the beginning of every prompt identical from one turn to the next, which is the property that a prefix cache rewards. This is my own inference, but it suggests that a design built for replay and recovery may also keep the cache-read costs from Diogo’s pie chart under control.

Each agent gets a computer that can be recreated quickly

The second part covered the computer that an agent works on. Lovable creates a base computer once and saves it as a golden checkpoint. From that checkpoint, it can recreate an agent’s computer environment in about 230 milliseconds, and the saved state includes files, memory, and running processes. Drafts live in shared memory. I did not capture the details of how that part is implemented.

A fast restore changes how teams can use agents. When an environment costs a fraction of a second to recreate, it becomes reasonable to give each task a clean computer, to discard one that has gone wrong, and to start several attempts from the same known state.

Verification happens in two loops

The third part described verification as two loops. The inner loop is the “fix it now, improve it” loop, which runs during the agent’s turn while the agent checks and corrects its own work. The outer loop covers what happens outside the agent loop, such as product metrics, visitor behavior, and business metrics like the number of transactions. The inner loop tells the agent whether the work is correct. The outer loop tells the team whether the work achieved what the business wanted.

The phrase “outer loop” also appeared in the morning, when Lauren Tan described Grok Bot as the outer loop around coding agents. She used it for workflow management, while Fabian used it for business outcomes, but both talks agree that a check inside the agent’s turn is not enough on its own.

What I am taking home

Diogo and Fabian describe two halves of the same job. Diogo argues that automation needs models that report calibrated confidence and software that breaks a task into stages that can be verified. Fabian shows the infrastructure that such software needs underneath it, namely a task history that survives crashes, environments that can be recreated on demand, and verification that reaches all the way to business metrics. For most Philippine teams, I would suggest three first steps:

If you would like help designing the decision stages, the verification, and the runtime for an agent workflow, our Claude Professional Services team can work through it with you, starting from the one decision that would make the biggest difference to your operations.