KitCutOur AI reviewer had perfect taste and only found small things
We gave an AI the job of reviewing our films before a person sees them, and asked for "a reviewer with perfect taste". It reviewed six times. Each time it found about two small things, and we fixed every one. None of the six rounds asked whether the film opened the right way. It didn't, and it took a person one look to say so.
KitCut makes whole films with Claude: the script, the pictures, the animation, the voice and the music. A film that is technically fine can still be wrong as a film: the wrong opening, too slow, a joke in a serious piece. So a second agent reviews every film before anyone else sees it. This post is about how that reviewer first went wrong, and the rule it follows now.
Six rounds, twelve small fixes, one wrong opening
The film was the second episode of Padel Tactics, our YouTube channel of animated padel explainers. Its opening wasn't working, so we built a review skill and ran it in a loop until the opening was good.
The first version took its job seriously. A fresh reviewer, one that had never seen the film being made, got four frames for every second, a list of measurements and seven scores to fill in. Each one came back with about two precise, timed notes: a label that stayed a beat too long, a frame held too long, a line of narration that could go. We made every change and cut the narrator's line. Then we went further and measured the reviewer against an earlier episode the owner had approved, and treated that episode's level as the target.
Six rounds later the owner watched the result and said the opening should be more like the first one we made. That was the real problem all along, and no round had come near it. Film 1 opens on the series' title painted over the court from the very first frame, makes its claim in two short lines, shows what the film covers and where its facts come from, and starts teaching at 0:25:
The opening we had spent a night on, about four and a half hours of it, was thrown away. The version built on film 1's pattern took 25 minutes.
It wasn't the only time
Once we looked, the same thing showed up wherever an AI judged taste:
- A template that passed its tests. A meetup invitation scored 4 out of 5 in three blind tests. Its source film was for a speakeasy-themed event (art deco, bottles), and that one-off theme became the costume of every remake. The owner's verdict: cringe. The grader had checked each film was well made. It never asked whether a developer meetup should look like a 1920s bar.
- A sponsor card that broke the sponsor's rules. A template for event sponsors stamped the sponsor's logo into a metal coin. It passed our checks and two test remakes. It also broke the brand's own standards, which ask for the logo flat, on a plain ground, with clear space around it. Nobody had opened the style book. (More on that in Real logo, real map, real people.)
- A 17-minute film that stood still. Two fresh reviewers passed a long research film. The owner stopped it in the first seconds: "It's too still. The video starts with a single still that doesn't move." Both reviewers had read still frames, one a second, and a still frame can't show you that nothing moves.
Why an AI critic finds small things
None of this is the model lacking taste. It's what we asked for.
- Asked for faults, a reviewer returns faults. A misplaced label is concrete, timed and easy to defend. "This is the wrong opening" is a judgement, and a reviewer told to list fixes will rarely write it.
- Small fixes look like progress. Each round found two new things, we fixed them, and the film got slightly better. When every round finds "about two things, somewhere new", it isn't converging. It's polishing.
- An approved film isn't the ideal. The owner had let that earlier episode through. They never said it was the standard. Measuring the new film against it turned "good enough to publish" into a target. As the owner put it: "i don't want you to consider the approved one an ideal. i never said it should be."
- Stills can't show motion. A sheet of frames shows composition and mistakes. It can't show pace, and it can't show that nothing on screen is moving.
The review we run now
The owner's instruction, in their words, is the rule: "its goal not to provide feedback that require micro changes but see the entire picture." A review now answers one question about the whole piece: would the person who asked for this, and a viewer who came for the subject, be glad this is the film?
- A verdict: Show it, Not yet, or Wrong idea. "It works. Show it." is a complete review.
- The whole in one paragraph: what the film is, and whether it does its job.
- At most three big changes, each one a viewer would notice: an order, a length, a picture, a tone.
- At most three plain mistakes, the kind anyone would see. In a padel film that's a player with no racket.
- No scores and no lists of small improvements.
Around it:
- Look at the model first. When the owner names a film as the model, lay the two openings side by side as one picture each (like the sheet above) before building anything. That one look would have saved the six rounds.
- Three rounds at most. A third Not yet means the idea is wrong, not the details.
- A fresh reviewer, but not a blind one. It never made the film and knows nothing of its history. It does get the owner's request in their own words, because "is this good?" means nothing without "good for what?".
- Watch motion in motion. Pace and stillness are measured on the video itself, not judged from stills.
- A reviewer's pass isn't approval. The owner's verdict is written down the same day, in their words, so the next film starts from what they actually said.
If you build an AI review loop
Ask for a verdict on the whole before asking for fixes, cap the number of changes, and give the reviewer the request it is judging against. If your loop keeps finding two small things every round, stop it and look at the whole thing yourself. And be careful which example you hold up as "good": the one you approved is just the one you let through.
Every film KitCut makes is also watched by a fresh reviewer for glitches before it reaches you, and we hold our own channels' films to the review above. When you want something changed, you can leave notes on a finished film and get its next version: see Changing a finished film. We wrote about another way a whole studio of films can quietly go wrong in Delete the style from your system prompt, and about the padel series itself in how KitCut made an 8-minute film from one brief.