Blog · 14 October 2026 · The KitCut team

We mined our AI agent's own transcripts. Here's what it bought us

Every KitCut film is made by an AI agent that writes the film in code, and every one leaves a full transcript of how it was made. After three busy days we had 172 films' worth. We read them all to find out where the agent's time went and what it kept building from scratch. Then we tested the fixes honestly, which meant claiming less than we wanted to.

KitCut turns a sentence into a finished animated film. Claude Opus is the director: it writes the narration, draws or paints each scene as JavaScript, records the voice, scores the music, looks at stills of its own work and fixes what it sees. That's a long conversation with a lot of tool calls, and each one is saved.

The question came from the founder, a few days after launch: "Should we regularly do this, analyzing the sessions and contributing to the core engine?" We wrote a script to do it and pointed it at the 172 films made between September 28 and 30: 234 Claude sessions in all.

Where the agent's time goes

First, a surprise about the whole film. Across those sessions Claude spent 27.1 hours composing, against 13.7 hours of tools running: rendering, recording, mixing. The machine work we had been trying to speed up was the smaller half.

Within Claude's own time, one step dominated:

The second surprise: only about 10% of what Claude writes is code. The rest is thinking: working out the design before the code exists. The first draft of a 30-second film's picture is one reply of about seven minutes, most of it spent reasoning. That's where the time goes, so that's what we set out to shorten.

One more number pointed the same way. A film remade from one of our templates took a median 5.6 minutes of Claude, against 13.7 for a 30-second film from scratch. A good starting point more than halves the work.

What it kept building by hand

Seven agents read the code of 69 of those films and listed every component Claude had built by hand: 1,476 of them. Many were the same thing over and over, in unrelated films: a pill-shaped label, a phone screen, a bar chart that grows, a logo lock-up, an end card, a soft glow, a page that slides away.

Whatever kept coming back across unrelated films went into a shared library, the kit: about 60 pieces covering text, charts, screens, icons, logos, end cards, transitions and timing cues, in every one of KitCut's looks. The agent gets only a short reference to it, not the code, so it costs almost nothing in the prompt.

The narrator was being set up to fail

The transcripts showed a second, cheaper problem. Our instructions asked the agent to write narration at 2.2 words per second of film. Real narration, with its pauses and breaths, runs at about 1.78. Claude writes exactly to the budget it's given, so its scripts were too long by design, and 53 of 71 first recordings ran past the end of the film and had to be recorded again.

We moved the budget to 1.7 words a second. About 57% of first recordings now fit as recorded, against 25% before.

We also tried a stricter fix: refusing a script before recording if it looked too long. Tested on the same 71 films, it would have refused 14 of the 18 that actually fit, because real speaking pace varies by about 20% with the voice and the take. We didn't ship it. The simple budget change did the job.

The same read found the narration checker crying wolf: numbers heard as digits ("1986" for "nineteen eighty-six") and names spelled another way counted as misreadings. That was 33 of the 78 lines it flagged. It now forgives exactly those.

Then we measured it, including the noise

A change to the studio touches every film, so we tested it with a bake-off: six real prompts from those three days, 30 seconds each, made four times on the same machine under the same load:

• Before, run 1, and before, run 2: the old code, made twice on purpose to see how much the numbers move on their own;
• Kit: the kit mentioned once, at the end of the instructions;
• Kit, in every step: the kit named wherever the agent actually draws.

What it says, honestly:

• The same code, run twice, took 90.5 and 80.2 minutes. That's about 11% of noise between identical runs, as large as the total gain we hoped to show. So six films can't prove the kit made whole films faster, and we don't claim it.
• The first draft of the picture is about 30% shorter. This part is clear: 26.4 minutes against 35.4 to 41.0.
• Mentioning a tool once isn't enough. With the kit named only at the end of the instructions, four of the six films still drew their own labels, phones and glows. Named where the agent actually draws, all six used it.
• Fewer recordings: 15 against 21 to 25, from the word budget and a guard that catches a broken take before it's used.
• Quality held. Both kit runs graded 4.00, between the two runs of the old code.

We shipped both changes, and the speed claim stays at what we measured.

What's still in the transcripts

The read found more than we fixed that week. A few we measured and left open:

• The music loop. The sound step already lowers the music under the voice, and then the agent asks for it lower anyway: in 10 of 19 narrated films.
• Copy and paste between scenes. On an 8-minute film written scene by scene, 10 of 12 scene sessions pasted the same 1.7 KB renderer, because one scene can't see what another wrote.
• Series re-reading themselves. Episodes of a series re-read the last episode's whole code in 8 to 10 films, where a short summary would do.

We rerun the harvest every few days. Each pass is a list of what the agent keeps doing by hand, and each fix gets the same test: two runs of the old code first, so we know how big the noise is before we believe a gain.

If you run an agent in production

• Its transcripts are the cheapest research you have. We found the wrong word budget in one afternoon. No user would ever have reported it; they would only have noticed films taking longer.
• Look for what it builds by hand over and over. That's your missing library.
• Put tools where the agent acts, not in a list at the end of the prompt.
• Run the old version twice before you measure the new one. If you skip that, an 11% swing looks like an 11% win.

How the agent fits into the whole process, from your sentence to the video, is on How a film is made. Two other things we learned by reading what the agent actually did: why it drew every film as a cartoon, and why our AI reviewer only found small things. The same habit of measuring before believing is behind Opus vs Sonnet: the cheaper model wasn't half the price and why long films are written scene by scene.

Make a film