The agent loop, and what it can't do in a browser

A tool-using agent is a much smaller idea than the discourse suggests. Send the conversation and a list of tool schemas to a model; if it asks for a tool, run it and send the result back; repeat until it stops asking. That is the loop, and it fits comfortably on one screen. This post is the whole thing: the twelve lines that are the agent, the two wire formats that exist in the world and the adapter that hides them, why every tool error goes back to the model instead of to the user, what could not be ported into a browser and what replaced it, and where the key lives when there is no server.

The voice agent on this site is a browser port of pyagent, a small terminal coding agent. Porting it was a useful exercise precisely because it forces you to separate the loop — which moves over unchanged — from the tools, which mostly cannot come at all.

The loop

messages.push(user_input)
for step in range(max_steps):
    reply = provider.complete(system, messages, tool_schemas)
    messages.push(assistant(reply.text, reply.tool_calls))
    if not reply.tool_calls:
        return reply.text
    for call in reply.tool_calls:
        result = run_tool(call.name, call.args)
        messages.push(tool_result(call.id, result))

That is the entire agent. Everything else is plumbing. Three details do real work.

The step cap. Without it, a model that misreads a tool result can loop until your budget is gone — and the failure is not exotic, it is a model calling streak_board repeatedly because the first result didn't contain the name it expected. Six steps is plenty for a question a person asked out loud, and the cap should be a small number chosen for the task rather than a large number chosen for safety.

Errors go back to the model, not to the user. When a tool throws, the exception is caught and returned as the tool result:

try   { return TOOLS[name].run(args); }
catch (e) { return "error: " + (e && e.message ? e.message : e); }   /* surface to the model */

So error: unknown team 'Montrael' arrives as a normal tool result. The model reads it, corrects itself, and calls again — and because that costs one extra step inside the same turn, the user sees a slight pause rather than a failure. Surfacing that exception to the user instead would end the turn on a mistake the model could have fixed in one step. Write your error strings for the model: name the parameter that was wrong and, where you can, list the valid values.

All calls in a reply get run. Models emit multiple tool calls per turn and expect every one to come back before the next completion. Run them, append all results, then loop — skipping one produces a conversation the API will reject, because a tool_use block without a matching result is a protocol error rather than a missing optional field.

Two wire formats, and only two

pyagent's provider layer exists because tool calling is expressed differently by different APIs. Having implemented both, the split is cleaner than expected: there are two formats in the world.

OpenAI-compatible — which also covers DeepSeek, Gemini's compatibility endpoint, Ollama, vLLM, LM Studio, Groq and llama.cpp, since they all expose the same endpoint shape — returns a tool_calls array, and the arguments arrive as a JSON string you have to parse yourself. Results go back as messages with role: "tool".

Anthropic returns tool_use content blocks with the input already a structured object, and results go back as tool_result blocks inside a user message.

That last row catches people: Anthropic takes the system prompt as a request parameter, not as the first message, and putting it in the messages array is silently accepted and differently interpreted.

Normalise both into one internal shape — {id, name, args} going out, {text, calls} coming back — and the loop and the tools never learn which model is running. Here is the entire OpenAI-side adapter, both directions:

// response → internal
var msg = ((d.choices || [])[0] || {}).message || {};
var calls = (msg.tool_calls || []).map(function (c) {
  var args = {};
  try { args = JSON.parse(c["function"].arguments || "{}"); } catch (e) {}  /* a string here */
  return { id: c.id, name: c["function"].name, args: args };
});
return { text: (msg.content || "").trim(), calls: calls };

// internal → request
function toOpenAI(m) {
  if (m.role === "tool") return { role: "tool", tool_call_id: m.id, name: m.name, content: m.content };
  if (m.role === "assistant") {
    var o = { role: "assistant", content: m.content || null };
    if ((m.calls || []).length) o.tool_calls = m.calls.map(function (c) {
      return { id: c.id, type: "function", "function": { name: c.name, arguments: JSON.stringify(c.args) } };
    });
    return o;
  }
  return { role: "user", content: m.content };
}

The try/catch around JSON.parse is not defensive programming for its own sake — models do occasionally emit malformed argument strings, especially under sampling temperature, and an empty object that fails the tool's own validation is a recoverable turn while a thrown exception is a dead one. Fifty lines of adapter is the whole architectural payoff of the split, and it is worth it.

Adding DeepSeek costs one table entry

The claim that there are only two formats is testable, and adding a third provider is the test. DeepSeek speaks the OpenAI wire format exactly — same endpoint path, same tool_calls array, same JSON-string arguments, same role: "tool" results — so it needs no adapter at all, only routing:

var MODELS = {
  anthropic: ["claude-opus-5", "claude-sonnet-5", "claude-haiku-4-5"],
  deepseek:  ["deepseek-v4-flash", "deepseek-v4-pro"],
  openai:    ["gpt-4o-mini", "gpt-4o", "llama3.1", "qwen2.5-coder"]
};
var DEFAULT_BASE = {
  anthropic: "https://api.anthropic.com",
  deepseek:  "https://api.deepseek.com",
  openai:    "https://api.openai.com"
};
/* Which wire format a provider speaks. */
var WIRE = { anthropic: "anthropic", deepseek: "openai", openai: "openai" };

function complete(msgs) {
  return WIRE[cfg.provider] === "openai" ? openaiComplete(msgs) : anthropicComplete(msgs);
}

Three lines of data and one lookup. That is what a correct abstraction boundary feels like from the inside: the thing you thought was a provider integration turns out to be a base URL and a model name.

Why DeepSeek specifically: for a public demo where visitors bring their own key, per-token cost is a real constraint on whether anyone tries the thing at all. deepseek-v4-flash is $0.14 per million input tokens and $0.28 per million output, which is one to two orders of magnitude below the frontier models, and it calls tools reliably enough for a five-tool agent. A full turn here — system prompt, tool schemas, one or two tool results, an answer — lands around a couple of thousand tokens, which at those rates is a fraction of a cent per question rather than something you watch.

What could not be ported

pyagent's tools are read_file, write_file, edit_file, list_dir and bash. Not one of them can cross into a browser tab. There is no filesystem and no shell — and if there were, handing them to a model on a public web page would be an extraordinary thing to do.

The honest options were to fake the tools, or to find real ones. The tools it got instead query data this site already publishes:

So "who's the best bat today?" spoken aloud produces a genuine streak_board(top=3) call against genuine data from the forecasting pipeline, and you watch the trace happen. The loop is demonstrating itself rather than being illustrated.

Tool schemas are prompt engineering

The most consequential thing about a tool is not its implementation, it is its description — that string is the only thing the model has to decide whether this is the tool for the question. Compare:

The second names the situations in which it applies, which is what routing actually needs. Every tool here also normalises its own inputs — a team-abbreviation map turns "the Habs", "MTL" and "Montreal" into the same lookup — because the alternative is a tool that fails on a reasonable phrasing and burns a step recovering.

Voice, with the browser doing the work

pyagent shells out to ffmpeg for recording, Whisper for transcription and piper for speech. The browser has all three built in: the Web Speech API's SpeechRecognition for mic → text and speechSynthesis for the reply. A Web Audio analyser drives the ring around the microphone from the live input level, so you can see it hearing you:

var an = ctx.createAnalyser(); an.fftSize = 256;
src.connect(an);
(function tick() {
  an.getByteTimeDomainData(buf);
  var peak = 0;
  for (var i = 0; i < buf.length; i++) peak = Math.max(peak, Math.abs(buf[i] - 128) / 128);
  ring.style.setProperty("--level", peak.toFixed(3));      // CSS does the rest
  raf = requestAnimationFrame(tick);
})();

Whose key, and where it lives

The model call happens in the visitor's browser, with the visitor's own key, held in localStorage. There is no server in this architecture to proxy through, which means the key never reaches this site — and also means it is sitting in browser storage, readable by anything that can run script on the page. The page says so plainly and offers a forget key button. Use a key you're willing to rotate, scoped and rate-limited if your provider supports it.

Anthropic's endpoint additionally requires an explicit opt-in header before it will answer a browser at all, which is a good design — it makes the unusual thing look unusual:

headers: {
  "content-type": "application/json",
  "x-api-key": cfg.key,
  "anthropic-version": "2023-06-01",
  "anthropic-dangerous-direct-browser-access": "true"    // you are opting in, loudly
}

The alternative — my key in a serverless function, so visitors need nothing — would mean paying for strangers' inference and defending an open proxy. For a demonstration of a loop, bring-your-own-key is the honest trade, and DeepSeek's pricing is what makes it a trade a visitor will actually take.

Using it yourself

To point this loop at your own tools, three things change and one does not. Replace the TOOLS table — each entry is a JSON Schema plus a function. Replace the system prompt. Choose your provider and key. Leave the loop, the adapters and the error handling alone:

TOOLS.my_tool = {
  name: "my_tool",
  description: "One sentence on what it returns, then a sentence on when to use it.",
  parameters: {                                   // JSON Schema, as both APIs expect
    type: "object",
    properties: { query: { type: "string", description: "what to look up" } },
    required: ["query"]
  },
  run: function (args) {
    if (!args.query) return "error: query is required";   // errors are strings, not throws
    return JSON.stringify(lookup(args.query));            // return text the model can read
  }
};

Three rules that matter more than the code: return text the model can read rather than opaque handles; keep results small, because every tool result is re-sent on every subsequent step and a verbose tool quadratically inflates a six-step loop; and make errors instructive, because the model is the one reading them.

Demo mode, and why it exists

With no key at all, the page still runs: the tools execute for real against real data, and only the model is stubbed by matching the question to a tool. The trace you see is genuine, the answer is genuine, and the page labels the model as absent. Same principle as the embedded demo scene in the splat viewer and the synthetic checkpoint in the weights pages — a demo should never be an empty state, and the honest way to achieve that is to make the parts that can run, run.