katlog.orgdevelopment journal Demo catalogueDemo
AI001
Briefing the AI2 min read

A 32-point audit: how to check an AI agent’s work

An AI agent built the system from our description and reported that everything was done. A code review showed that a lot was lost and some things were only half done. How we reorganised the work so this wouldn’t happen again.

Alexander Ananyev

The prototype of the new catalogue and the real Katlog system are two different codebases. The prototype was built in a conversation; for everyday use we needed a server, a database and password login. We described everything in words, and an AI agent on a development platform wrote the system from scratch.

What got lost in the move

The first version looked different and worked only in Russian, even though the brief explicitly listed three languages. Later we found more: “Nearby on the shelf” didn’t work, the saved list couldn’t be exported to RIS, filters allowed only one value, and the “Data quality” section showed nothing but numbers. A full comparison of the code with the prototype produced 32 discrepancies.

Three surprises

  1. Tasks didn’t startThe agent was waiting for confirmation in the development platform’s interface. Our requests arrived, but without the owner’s approval they never entered the work queue.
  2. The agent lost the listIn the middle of a large job it asked for items 1–32 again: the numbering had vanished from its memory.
  3. “Done” doesn’t mean “done right”The subject filter was “finished”, but the server didn’t know the “ - ” separator used in all our headings. Language names were built for codes like “en”, while the catalogue uses “eng”.
The file docs/katlog-fixes.md in the project: the brief saved verbatim
The first step of every brief is to save it verbatim to a project file. Below the text, the agent keeps a table: item, status, where in the code, how it was checked. The screenshot shows the Russian original.

How we work now

  • The first step of every task is to save it verbatim to a file in the project. For example, docs/katlog-fixes.md. The list isn’t lost even if the agent forgets the context.
  • A table with a mark for every item: done or not, where in the code, how it was checked.
  • Verify the code, not the report. Especially where it is easiest to fake: appearance settings, permissions, counts.
  • A task no longer than fits in one message. The agent has a length limit; big lists are split into parts.
  • A live test by the librarian. The bug with record deletion was found not by a test but by ordinary work with the system.
The key pointAn AI agent is a strong executor but tends to simplify large tasks and silently skip details. What it needs is not trust-based supervision but a clear list and a check of every item.

What a task written by these rules looks like is shown in “A complete brief: ISBN lookup”.

The good news

Once the process settled, the agent’s reports became honest. In the latest one it listed separately what it couldn’t verify: working with a barcode scanner and the full path in a browser. A report like that is more useful than “all done”.