Agent learning notes: make a run inspectable

Make the first run observable

The exercise starts with a terminal trace rather than a chat box: retrieve records, propose a draft, await approval and return the result. This separates a statement about saving from a tool that actually stored something.

Every tool result carries a callId, including errors. No matches produce an empty array; invalid arguments produce INVALID_ARGUMENTS. Callers do not need to infer machine state from prose.

{ "callId": "call-1", "content": { "ok": true, "data": [] } }
{ "callId": "call-2", "content": { "ok": false, "error": "USER_DENIED" } }

Two tools are enough to reveal the boundaries

Search and draft saving are the only tools. Search reads three fixtures; saving accepts only retrieved IDs. That does not establish relevance, but it prevents invented IDs from masquerading as retrieved sources. Relevance still needs retrieval and answer evaluation.

There are business checks beyond types: whitespace is not a query, duplicate IDs should not create duplicate reading tasks and approved:true is not an allowed argument. Schema declarations do not remove the need for execution-time validation.

Approval must refer to one exact proposal

A write proposal is snapshotted before terminal confirmation. One test mutates the preview object and then approves it; the stored title and IDs remain the originals. That is why snapshot returns a deep copy.

Expiry, mismatched IDs and repeated decisions have explicit outcomes. Refusal returns USER_DENIED rather than becoming a network failure that should be retried automatically. Recovery needs to distinguish a person’s decision from a technical failure.

A web version needs authenticated users, sessions and authorization. The terminal exercise has no account system and its random approval ID is not an authentication mechanism.

Late results must not resume a stopped run

A late stream chunk can overwrite newer UI state. In an agent, a late result could also propose another action. The runner bounds model waiting with cancellation and timeout, then checks the signal again before consuming a result.

A test model deliberately ignores abort. After the host stops waiting, it returns save_draft; no approval or draft appears. This verifies rejection of late output, not termination of the background task.

A session rejects overlapping run calls and accepts decisions only while awaiting approval. State-layer checks remain effective under rapid clicks or additional callers.

What was checked

All 18 tests passed. They cover the normal path, denial, unknown tools, extra fields, unretrieved IDs, mutated previews, duplicate and expired approvals, deduplication, limits, cancellation and late output. Deterministic scripts and events avoid dependence on a model happening to emit a particular answer.

The tests do not cover real retrieval quality, model factual accuracy, provider rate limits or cross-process recovery. State vanishes on exit and deduplication does not extend across sessions. Persistence and authorization tests are additional work.

A test record next to the code is more useful to a reader than broad reliability claims. It shows what happens when an action is refused or interrupted.

Exercises for the next iteration

• Replace search_notes with retrieval over an explicitly selected public notebook directory, retaining paths and passages. Test missing results and duplicate names before adding vectors.
• Add one real-model adapter. Keep a fixed task set and model settings, record inputs, tool traces and failures, and apply redaction rules to user content and credentials.
• Connect it to the React and Ant Design exercise with a pending-approval view showing changes, sources and cancellation. Check mobile controls and keyboard focus.
• Consider an orchestration framework such as LangGraph once persistence, resumable pauses or branching justify it. Framework selection does not replace tool authorization or task evaluation.

The next learning goal is to explain one action clearly: its source, its authorization, how many times it ran and what remains after failure.