All posts

Engineering

8 min

What 2M runs taught us about retries

Retries look free until they double your token bill. Notes from running two million agent runs a week.

Sam Okoro

Co-founder, Eng

Every agent demo has the same moment: the model does something impressive, and somebody in the room asks, “but what if it gets it wrong?” For most teams, that question kills the project.

Our answer is boring on purpose. Agents run on their own until they reach a line you drew. Past the line, a person decides.

The 4% problem

If an agent asks for approval too often, people stop reading the requests. Across two million runs a week, approvals stay meaningful when fewer than 4% of runs need one.

An approval nobody reads is worse than no approval at all — it’s a rubber stamp with your name on it.

Where to draw the line

We start every customer with three rules and tune from there: a dollar threshold, a list of new or unusual recipients, and anything that can’t be undone.

Share

Create a free website with Framer, the website builder loved by startups, designers and agencies.