Skip to article
The Design Office of Nick Sloggett
WorkWritingExperimentsResumeEmail
Back
Essay·Nov ’24·3 min read

Agents will use your app the way a screen reader does

AI agents can now operate interfaces built for people. The ones that work best for them will be the ones already built for screen readers.

Typeface

AI agents have started using software the way people do: looking at the screen, moving a pointer and typing. The apps they’ll handle best are the ones already built for screen readers, because an agent and a screen reader need the same thing from an interface: every control named, every state in words.

How does an AI agent use an app?

On October 22, Anthropic released computer use in public beta: Claude operates a computer by looking at a screen, moving a cursor, clicking buttons and typing. It works from screenshots. To click something, it counts how many pixels it needs to move the cursor. Anthropic is candid that it’s slow and often wrong. Google is reportedly building something similar into Chrome, also driven by screenshots.

What is the Model Context Protocol?

Yesterday Anthropic released the Model Context Protocol, an open standard for two-way connections between AI assistants and the systems where data and tools live, starting with servers for Google Drive, Slack, GitHub and Postgres. It’s the structured way in: a product describes what it can do, and an assistant calls it directly.

Where a product offers a structured way in, an agent will use it. Where it doesn’t, the agent falls back to pixels, and most software won’t have a structured way in for years.

Pixels are the worst interface for a machine

A screenshot is a picture of an interface. To act on it, a model has to guess what each shape is, whether it’s enabled and whether the press worked, and the guesses compound across a task. An agent working from structure (a button with a name, a field with a label, a checkbox with a state) reads what the interface says about itself.

OSWorld, a benchmark of real computer tasks published in April, gave agents the screen four ways. In its own runs, the best result was GPT-4 reading the accessibility tree with no screenshot at all: 12.24% of tasks, against 72.36% for people. Claude now scores 14.9% from screenshots alone, so pixel-reading is catching up. Every agent still fails most tasks, and the paper names the main weakness: matching what it sees to the control it means.

Screen readers got there first

Assistive technology has asked for exactly that structure for decades. On Android, TalkBack can only announce a control if the control declares what it is. A button without a content description is announced as an unlabeled button, and the listener has to guess the rest. A custom toggle that only changes color has no state at all to anyone who can’t see it.

Every app that skipped that work made itself harder to use for blind people. The same shortcuts now make it harder to use for agents.

What to design for

This is the accessibility work most teams have been deferring.

  • Name every control by what it does, in words a person would use.
  • Put state in text: selected, disabled, loading, failed. Color alone tells a machine nothing.
  • Say what a destructive action will do before it does it, in the interface rather than in a toast afterwards.
  • Make undo a real control, so a mistake by a person or an agent has a way back.

A test anyone can run: turn on your platform’s screen reader, close your eyes and complete your product’s most important task. Where you get stuck, an agent will too.

I’ve spent the last few years on Android designing one screen for many people, read from across a room or up close. The lesson from that work applies here. An interface that only works for one kind of reader is a fragile interface. Agents are about to become one more kind of reader.

How does an AI agent use an app?What is the Model Context Protocol?Pixels are the worst interface for a machineScreen readers got there firstWhat to design for
Previous← Generation grew a timelineNextThe thing gets its own page →

© 2026 Nick Sloggett

EmailLinkedIn (opens in new tab)ResumeRSS