Back
EssayMar ’242 min read

Design for the other 86 percent

Cognition says Devin resolves 13.86% of real GitHub issues on its own. That’s a big jump, and it means most of the product is what the agent hands back when it doesn’t finish.

Typeface

On March 12, Cognition introduced Devin, which it calls “a tireless, skilled teammate.” Devin works in a sandbox with its own shell, code editor and browser. On SWE-bench, a set of real issues from open source projects, Cognition says it resolved 13.86% of a random sample end to end. The previous best it cites was 1.96%.

That’s a real jump, roughly seven times the old number. It also means that on about six issues out of seven, the agent didn’t finish.

Most of the reaction has been about the 13.86%. I keep thinking about the rest. When an agent finishes one task in seven, the product is what it hands back on the other six.

A failed attempt by a person comes with context. You can ask what they tried, where they got stuck, and what they’d do next. A failed attempt by an agent comes with whatever the interface chose to keep. If that’s a closed ticket and a wall of logs, the engineer who picks it up starts from scratch, and the agent cost them time instead of saving it.

The design choice I find most interesting in the announcement is the workspace. Devin uses the tools an engineer already uses, a shell, an editor and a browser, so its work happens in places a person knows how to read. That treats the agent’s process as something someone will need to pick up. It’s the right instinct.

I spent two years at Uber designing tools for engineers, and the one I think about most is Studio. It let an engineer run a trip through every state from a desk, with the map on top and the stream of events underneath. Studio didn’t fix anyone’s bug. It showed exactly which state the trip was in when it broke, and that was most of the fix.

Agents need the same thing. On a task it didn’t finish, the handback should answer three questions in the first screen:

  • What was it asked to do, in its own restated words?
  • How far did it get, and what did it change?
  • Where did it stop, and why?

If those answers are clear, a 14% success rate is a useful colleague. If they aren’t, it’s a slot machine.

One caution about the number. A benchmark is a set of tasks someone chose because they could be scored. Real work arrives half-specified, with a deadline and a person waiting. I expect agents to get much better at the first kind of task quickly. The second kind will depend on how well they hand back what they couldn’t do.