Blog · 25 October 2026 · The KitCut team

How to test AI image generators automatically, with Claude as the judge

To test AI image generators automatically, write prompts whose facts can be checked, turn each fact into a yes/no question, and have a vision model answer the questions blind, with OCR reading the text. We ran 12 such prompts through 8 image models: 96 pictures for $4.31. Two Claude Opus 5.5 judges answered 304 questions each without knowing which model made which picture, and agreed on 291 of 304 answers; a third judge settled the 13 they disputed. Below: the pipeline, the code, the costs, the disagreements and the limits.

Updated 25 October 2026. The pictures, the judges' answers and every number below are from that day.

At a glance

• The method: 12 prompts and 48 checks per model: 38 yes/no questions for a judge, one fact each, and 10 for OCR (nine strings it must find, one picture that must have no text).
• The judges: two blind Claude Opus 5.5 judges, two runs of the same model, agreed on 95.7% of 304 answers (Cohen's kappa 0.72). Eight of their 13 disagreements were one question about fingers that they read in two ways. Agreement shows consistency, not truth: we have not yet checked the answers against people.
• OCR against the judges: 10 of OCR's 11 failures were words run together where the lettering has the space (“10AM” for “10 AM”), so we now ignore spaces, and OCR and the judges agree on 22 of 24 text facts, up from 18.
• People: every model passed every check on the two children and the fisherman; hands were harder to draw and to judge: 7 of 8 pictures showed exactly two hands, and none drew the piano's black keys right.
• Cost: $4.31 for the pictures, $0.045 each. Judging them twice would cost about $2.89 through the API, or $1.44 with the Batch API, on our estimate.
• The limits: it checks facts, not taste, and it cannot judge motion in video.

How do you evaluate an AI image generator?

There are three ways to evaluate text-to-image models:

• People vote between two pictures, as in the arenas. That measures preference and takes thousands of votes.
• A learned score such as CLIPScore, PickScore or ImageReward rates each picture. It is cheap, but some image models are trained against these same scores and learn to please them.
• A checklist breaks the prompt into facts and asks a vision model one yes/no question per fact: the method of TIFA (2023) and Davidsonian Scene Graph (2024). The original GenEval (2023) checked objects, counts, colours and positions with an object detector instead.

The checklist has the best evidence. GenEval 2 (December 2025) scores pictures with Soft-TIFA, which has an open vision model, Qwen3-VL-8B, answer one question per fact. Against people's labels it reached an AUROC of 94.5%, ahead of VQAScore (92.4%), TIFA (91.6%) and VQAScore on GPT-4o (91.2%); AUROC is the chance that a score ranks a picture people marked right above one they marked wrong. The paper also found the original GenEval had drifted from human judgement by up to 17.7 points on a recent model.

This is LLM-as-a-judge for images, also called VLM as a judge. Ours is a small checklist bench with Claude Opus 5.5 as the judge, built so that a person can check every step.

Step 1: write prompts whose facts can be checked

Every fact a prompt states becomes a check: a yes/no question for the judge, or words OCR must find. The portrait prompt from our spec:

{
  "id": "p09-portrait",
  "ability": ["binding", "photoreal"],
  "prompt": "Photorealistic portrait of an elderly fisherman with a grey beard, wearing a yellow raincoat, holding a silver fish, a harbor with boats behind him, golden hour light.",
  "checks": [
    {"id": "j-1", "type": "judge", "q": "Is he an elderly man with a grey beard?"},
    {"id": "j-2", "type": "judge", "q": "Is he wearing a yellow raincoat?"},
    {"id": "j-3", "type": "judge", "q": "Is he holding a silver fish?"},
    {"id": "j-4", "type": "judge", "q": "Is there a harbor with boats behind him?"},
    {"id": "j-5", "type": "judge", "q": "Does it look like a photograph rather than a painting or render?"}
  ]
}

That is one of twelve. The whole spec, with every prompt and check, is image-bench-2026-10-spec.json (10 KB), and the rules our judges followed are judge-instructions-2026-10.md (2 KB).

Text checks are ocr_contains (these words must be read off the picture) or ocr_none (no text at all). The twelve prompts cover eight abilities:

What we learned writing the questions:

• One fact per question. The girl's coat and her umbrella are two checks, so you can see which one a picture got wrong.
• Ask about everything you care about. Both judges noted, unasked, that one model drew teenagers rather than young children. No check asked about age, so it did not count.
• Keep counts small. Anthropic's vision guide says Claude's counts are approximate, “especially with large numbers of small objects”; ours asks for four apples and two pears.
• Tag abilities per check, not per prompt. Our spec tags whole prompts, so the question about the piano's black keys counts as anatomy, with the two about hands. No picture passed it, so our anatomy column says as much about keyboards as about hands.

Step 2: generate under a spending cap

All eight models are called through one API, OpenRouter: a square picture of about one megapixel, medium quality where offered, one per prompt. --plan prices the run first. During --run, each request reserves its estimated price, and one that would pass the cap is never sent:

def one(item):
    k, m, p, path = item
    est = estimate(m, spec) or 0.05
    with lock:
        if spent_now() + reserved[0] + est > cap:
            return "skipped (cap)"
        reserved[0] += est
    ...  # make the picture, record the cost the API reports, release the reservation

The spend is what the API reports per finished picture: $4.31 for the 96, against a $5.20 cap. The API key has its own limit too: when we tried four harder prompts later that day, it refused them all, and nothing was charged.

Step 3: read the text with OCR

For exact words, a deterministic reader is the first check. We run RapidOCR 1.2.3, with its default models, on every picture, keep lines read with confidence of 0.5 or more, and normalise both sides:

def norm(s):
    s = unicodedata.normalize("NFKC", s).upper()
    return " ".join("".join(ch if ch.isalnum() else " " for ch in s).split())


def ocr_verdict(check, reading):
    joined = norm(reading.get("joined", "")) if reading else ""
    if check["type"] == "ocr_contains":
        # spaces do not count: RapidOCR runs words together ("EVERYFRIDAY", "10AM") on lettering
        # that has the space, which was 10 of the 11 OCR fails in the October 2026 run; the
        # judge's spelling check reads the lettering itself
        return norm(check["expect"]).replace(" ", "") in joined.replace(" ", "")
    if check["type"] == "ocr_none":
        real = [
            x
            for x in (reading or {}).get("lines", [])
            if x["score"] >= 0.6 and len(norm(x["text"])) >= 2
        ]
        return not real
    return None

Case, punctuation and spaces are ignored. Spaces counted in our first rule; the OCR section below shows why that changed. The no-text check counts a line as text when it has two or more characters read with confidence 0.6 or more. RapidOCR's default models read Chinese and English, not Cyrillic, so our Ukrainian chalkboard is checked by the judges only.

Step 4: blind the pictures

For each prompt, the packet step shuffles the models, gives the pictures fresh letters, saves each as a JPEG of at most 1024 pixels, and writes the letter-to-model key to a separate file:

def packets(spec, seed=20261025):
    ...  # root is the bench folder, pdir its packets/ folder
    rnd = random.Random(seed)
    key = {}
    letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    for p in spec["prompts"]:
        models = [
            m
            for m in spec["models"]
            if os.path.exists(os.path.join(root, "images", slug(m), p["id"] + ".png"))
        ]
        rnd.shuffle(models)
        imgs = []
        key[p["id"]] = {}
        for i, m in enumerate(models):
            lab = letters[i]
            key[p["id"]][lab] = m
            src = os.path.join(root, "images", slug(m), p["id"] + ".png")
            dst = os.path.join(pdir, "%s-%s.jpg" % (p["id"], lab))
            with Image.open(src) as im:
                im = im.convert("RGB")
                im.thumbnail((1024, 1024))
                im.save(dst, "JPEG", quality=90)
            imgs.append({"label": lab, "path": dst})
        checks = [{"id": c["id"], "q": c["q"]} for c in p["checks"] if c["type"] == "judge"]
        save_json(
            os.path.join(pdir, p["id"] + ".json"),
            {"prompt": p["id"], "text": p["prompt"], "checks": checks, "images": imgs},
        )
    save_json(os.path.join(pdir, "key.json"), key)

A packet holds one prompt's text, its judge questions and its lettered pictures (paths shortened, two of eight shown):

{
  "prompt": "p06-binding",
  "text": "A girl in a red coat holding a green umbrella stands next to a boy in a yellow raincoat holding a blue umbrella, on a rainy street.",
  "checks": [
    {"id": "j-1", "q": "Is the girl wearing a red coat?"},
    {"id": "j-2", "q": "Is the girl holding a green umbrella?"},
    {"id": "j-3", "q": "Is the boy wearing a yellow raincoat?"},
    {"id": "j-4", "q": "Is the boy holding a blue umbrella?"}
  ],
  "images": [
    {"label": "A", "path": "packets/p06-binding-A.jpg"},
    {"label": "B", "path": "packets/p06-binding-B.jpg"}
  ]
}

• New letters for every prompt. If a model were always picture C, a judge could form an impression of C and carry it across twelve prompts; fresh letters and a fresh order break the link. The fixed seed makes the shuffle repeatable, so re-running it keeps every verdict valid.
• The key lives apart. The judge instructions say: “Do not look in the images/ folder, results.json or key.json.” Those hold the pictures and costs under each model's name.
• No guessing. The judge “must not guess or say” which model made a picture. Letters hide names, not styles, and a judge that names makers invites the bias blinding removes. It helps that the judge has no stake: Anthropic sells no image generator, and its documentation says Claude “cannot generate, produce, edit, manipulate, or create images”.

The size limit also fixes the bill. Claude counts one token per 28-by-28-pixel patch, so each packet picture is 1,369 input tokens.

Step 5: LLM-as-a-judge, run twice and blind

Our two judges were Claude Opus 5.5 agents inside Claude Code, not raw API calls. Each got the packets and the same one-page instructions, opened every JPEG and wrote one JSON answer per picture per check, without seeing the other's answers. From the instructions:

Look properly: zoom in mentally on small text, count objects one by one, check fingers and keys individually, compare colours to the words.

"yes" only when the picture clearly satisfies the question as written. "no" when it fails, when it is ambiguous, or when you cannot tell. Spelling must be exact for text checks (a missing, extra, doubled or wrong letter is "no"; ...)

Give a short note (under 20 words) for every "no", and for counts.

The rule that an unclear picture gets a no makes “yes” mean “clearly yes”, and it caused most of the disputes below.

The same judge as an API call

The equivalent with Anthropic's official Python SDK sends one packet's pictures as base64 image blocks, then the checks, and gets the answers back as JSON matching a schema. Save it as judge.py; we compiled it but did not run it.

import base64
import json
from pathlib import Path

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

RULES = (
    "You are a strict judge of AI-generated pictures. Each picture has a letter; you do not "
    "know which model made it and must not guess. For every picture and every check, answer "
    "yes only when the picture clearly satisfies the check as written, and no when it fails, "
    "is ambiguous or you cannot tell. Spelling must be exact. Count objects one by one. Give "
    "a short note (under 20 words) for every no and every count."
)


def request(packet):
    """The Messages API body for one packet: the pictures first, then the checks."""
    content = []
    for pic in packet["images"]:
        data = base64.standard_b64encode(Path(pic["path"]).read_bytes()).decode()
        content.append({"type": "text", "text": f"Picture {pic['label']}:"})
        content.append(
            {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}}
        )
    checks = "\n".join(f"{c['id']}: {c['q']}" for c in packet["checks"])
    content.append({"type": "text", "text": f"Prompt: {packet['text']}\n\nChecks:\n{checks}"})
    answer = {
        "type": "object",
        "properties": {
            "label": {"type": "string", "enum": [p["label"] for p in packet["images"]]},
            "check": {"type": "string", "enum": [c["id"] for c in packet["checks"]]},
            "answer": {"type": "string", "enum": ["yes", "no"]},
            "note": {"type": "string"},
        },
        "required": ["label", "check", "answer", "note"],
        "additionalProperties": False,
    }
    schema = {
        "type": "object",
        "properties": {"answers": {"type": "array", "items": answer}},
        "required": ["answers"],
        "additionalProperties": False,
    }
    return {
        "model": "claude-opus-5-5",
        "max_tokens": 16000,
        "system": RULES,
        "messages": [{"role": "user", "content": content}],
        "output_config": {"effort": "high", "format": {"type": "json_schema", "schema": schema}},
    }


def read_answers(packet, response):
    if response.stop_reason != "end_turn":  # a refusal or max_tokens: re-queue, never guess
        raise RuntimeError(f"{packet['prompt']}: stopped with {response.stop_reason}")
    text = next(b.text for b in response.content if b.type == "text")
    answers = json.loads(text)["answers"]
    asked = {(pic["label"], c["id"]) for pic in packet["images"] for c in packet["checks"]}
    got = [(a["label"], a["check"]) for a in answers]
    if len(got) != len(asked) or set(got) != asked:  # a schema cannot fix the count
        raise RuntimeError(f"{packet['prompt']}: an answer is missing or repeated")
    for a in answers:
        a["answer"] = a["answer"].lower()  # enum capitals are not guaranteed
    return answers


if __name__ == "__main__":
    packet = json.loads(Path("packets/p06-binding.json").read_text())
    print(read_answers(packet, client.messages.create(**request(packet))))

• Pictures first, each with its letter. Anthropic's vision guide says Claude works best with images before text, and suggests labelling each one.
• Eight pictures a request. The API takes up to 600 images per request, but above 20 each must be at most 2,000 pixels a side. Eight 1024-pixel JPEGs come to at most 3.6 MB as base64 in our packets, well inside the 32 MB request limit.
• A schema, not a plea for JSON. Structured outputs (output_config.format) make the reply match the schema, so an answer can only be yes or no. Two things it does not promise: how many answers come back, and an enum value's capitals. So the code checks that every picture-and-check pair is answered once, and lower-cases the answers.
• A short note, not reasoning. The structured outputs guide warns that a field asking for step-by-step reasoning might trigger a refusal.
• Effort, not a thinking budget. On Opus 5.5 thinking cannot be switched off and budget_tokens is rejected; effort sets how much it thinks, and defaults to medium. We have not measured its effect on the answers.
• No fallback model. The API can retry a refused request on another model, but that would quietly change the judge, so a refusal stops the packet instead.

The same request bodies go through the Message Batches API at half the price. Most batches finish within an hour:

import json
from pathlib import Path

import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

from judge import request  # the function above

client = anthropic.Anthropic()

packets = [json.loads(p.read_text()) for p in sorted(Path("packets").glob("p*.json"))]
batch = client.messages.batches.create(
    requests=[
        Request(custom_id=f"{judge}-{p['prompt']}", params=MessageCreateParamsNonStreaming(**request(p)))
        for judge in ("judge-a", "judge-b")
        for p in packets
    ]
)
print(batch.id, batch.processing_status)  # poll until "ended", then read the results by custom_id

Step 6: reconcile the two judges

Where the judges agree, that is the answer; where they disagree, a third judge, another run of Claude Opus 5.5, answers just that picture and question, and its answer stands. Across the 304 paired answers:

They agreed on 291 (95.7%). Judge A said no 22 times and judge B 29; they said no together 19 times. All 13 disagreements sit in three prompts, and eight of them are a single question.

Agreement is easiest to trust where a question leaves nothing to interpret. On the poster, one model split the first line in two and both judges said so in almost the same words; OCR, which joins the lines it reads into one string, cannot see it.

What Cohen's kappa means here, and its limit

Raw agreement flatters when most answers are yes: two judges who always said yes would agree often without looking. Cohen's kappa (1960) subtracts the agreement their rates of yes would produce by chance. In our script, po is the observed agreement, pa and pb each judge's share of yes, and pe the agreement chance alone would give:

def kappa(pairs):
    """Cohen's kappa for two raters' yes/no answers [(a, b), ...]."""
    n = len(pairs)
    if not n:
        return None
    po = sum(a == b for a, b in pairs) / n
    pa = sum(a for a, _ in pairs) / n
    pb = sum(b for _, b in pairs) / n
    pe = pa * pb + (1 - pa) * (1 - pb)
    return round((po - pe) / (1 - pe), 4) if pe < 1 else 1.0

With our numbers: observed agreement 0.9572. Judge A said yes 92.8% of the time and judge B 90.5%, so chance alone would give 0.8460. Kappa is (0.9572 − 0.8460) / (1 − 0.8460) = 0.72, well short of the raw 95.7%, because most of the agreement was on yes, where chance does well. When one answer dominates, kappa can be low while agreement is high, a known paradox of the measure (Feinstein and Cicchetti, 1990), so we report both.

The bigger limit: our judges are two runs of one model reading the same packets. They share training and blind spots, so where one misses a detail the other may too, and agreement goes up, not down. It shows that a question can be answered consistently, not that the answers are true.

Only people can show that. In a calibration round, two people answer the same checks on a sample of pictures, every no and every dispute included, and you report how often the judge matches them per kind of check; kinds it gets wrong too often are marked low confidence or dropped. We have not run that round yet, so read our judges' answers as careful readings, not ground truth.

Where the judges disagreed, and what the third judge said

The judges disagreed on 13 answers, in five questions. After reading every note, we sort them into four kinds: nine come from how a question was worded, four from details that are hard to see.

• A question read two ways (8). “Does every visible hand have five fingers and plausible joints?” Judge A answered yes on all eight pictures: it saw no fused, extra or missing finger. Judge B answered no on all eight: on six, the players' poses hide some fingers, so it could not count five, and the rules say no when you cannot tell; on two, it suspected a gap or a misplaced thumb. The question asked two things at once.
• A pattern hard to count (2). Judge B passed both keyboards, though its own note on one reads “pair, pair, triple”, which breaks a piano's pattern of twos and threes too; judge A found two pairs or two triples in a row.
• A tiny shape far away (2). In two empty streets, a few pixels at the far end could be a person or a car: ambiguous to judge B, so no; judge A said yes.
• A borderline definition (1). One diagram's background is (246, 251, 253), a faint blue. Both judges put its colour within a point of that; one called it white, one did not.

The third judge answered yes on 11 and no on 2 (fingers, yes on 8; black keys, no on 2; far-away shapes, yes on 2; background, yes on 1), and its answers are the ones in the scores. For the fingers question it had one line the first two judges did not, taken from the check's own brackets: a finger hidden by the angle is not a missing finger. The lesson is about the questions more than the judges: ask what the picture shows, not what it hides. Our judge instructions now say so for every run:

Ask what the picture shows, not what it hides: a finger, hand or object hidden by the camera angle, by another object or by the frame is not missing. Where a check names the defects it is about in brackets ("no fused, extra or missing fingers"), answer "no" only when one of those defects is visible.

Wording would have settled the background too: “Is the background white or nearly white?” if a faint tint should pass. That is what a second judge is for: its disagreements point at the questions to rewrite.

OCR or the judge: which reads text better?

Three text facts were checked both ways on all eight pictures: GRAND OPENING on the sign, “Saturday 10 AM” on the chalkboard and KITCUT on a sneaker. With our first OCR rule, OCR and the judges agreed on 18 of the 24. All six disagreements were on the chalkboard, so we looked at each one:

• Four times, OCR dropped a space. On the boards from Seedream 5 Pro, Seedream 5 Flash, Nano Banana Pro and Grok Imagine 2.0 it read “10AM”. Each board shows a gap, narrow on two, and both judges said yes; our first rule kept spaces, so OCR failed all four.
• Once, OCR missed a word. Nano Banana 2.1 wrote “Saturday” large, in looping chalk script, on a board seen at an angle. OCR returned only “10 AM”.
• Once, the judges were stricter. GPT Image 2.5 Sunburst's board says “Saturday 10 AM.”, with a full stop. OCR ignores punctuation and passed it; both judges failed it, because the check says exactly “Saturday 10 AM”. Neither is wrong: they follow different rules, and the rule should be set before the run.

The space problem ran through the whole bench: 10 of the 11 OCR checks that failed under the first rule failed only on a space (“EVERYFRIDAY”, “6PMTO MIDNIGHT”), on posters and boards that show the spaces. So we changed the rule to ignore spaces and re-ran the scoring. OCR and the judges now agree on 22 of 24 (91.7%). One OCR check now fails in the whole bench: Nano Banana 2.1's chalkboard, where OCR finds no “Saturday”. The other difference left is Sunburst's full stop.

OCR has a second blind spot: on the empty street it found no text in any picture, while the judges found sign plates or letter-like marks in five. A blank sign is a sign, but not text.

OCR now checks the letters; spacing, punctuation and layout are the judge's job. OCR stays because it is free and repeatable, and because where it and the judge disagree is where a person should look: that is how we found this.

What does it cost to test AI image generators?

Making the pictures cost $4.31, or $0.045 a picture, from $0.0134 (GPT Image 2.5 Sunburst and Flare) to $0.1345 (Nano Banana Pro). Our judges ran inside Claude Code rather than through the API, so there is no per-request bill to report. At Claude Opus 5.5's published API prices of $4 per million input tokens and $20 per million output tokens, the same judging would cost:

• Pictures: a 1024-pixel square is ⌈1024/28⌉ × ⌈1024/28⌉ = 37 × 37 = 1,369 tokens. Two judges read all 96: 262,848 tokens, $1.05.
• Text and answers: we count 1,000 tokens of rules, prompt and checks for each of 24 requests ($0.10), and our judges' 608 answers average 73 characters of JSON, about 14,900 tokens at three characters a token ($0.30).
• Thinking: the unknown. Opus 5.5 decides how long to think, billed as output: each 1,000 thinking tokens per request adds $0.48. At 3,000 the total is $2.89; from 1,000 to 6,000 it runs $1.93 to $4.33. The Batch API halves it all ($1.44 at 3,000), and a third judge adds about $0.07 per disputed answer.

Through the Batch API, two judges cost about what one of the cheapest pictures does: $0.015 a picture against $0.0134. To spend less, send fewer pixels: a 2,048-pixel square is shrunk to fit the 4,784-token cap and still costs 4,761 tokens, three and a half times a 1,024-pixel one.

What this method does not measure

• Taste. A checklist asks whether the facts are there, not whether the picture is good; for preference, compare pairs. In GenArena (February 2026), a vision model judging pairs ranked seven image-editing models with a Spearman correlation of 0.86 against LMArena's human-vote leaderboard; scoring one picture at a time reached 0.36.
• Video. A judge sees frames, not motion. On the test set of VideoScore2 (September 2025), the scores GPT-5, Claude Sonnet 4 and Gemini 2.5 Pro gave generated videos correlated with people's at 0.18 to 0.26 on average, and 0.17 to 0.20 on physical and common-sense consistency. When we made an episode with MiniMax H3, every shot still had to be checked by eye (running MiniMax H3 on an 8 GB GPU).
• Variation. One picture per prompt is one draw; several turn each pass into a rate with an interval.
• Hard cases. 41 of our 48 checks were passed by all eight models and one by none; only 6 told models apart. A bench every model passes ranks nothing, and GenEval's drift shows how fast that happens.

What the first run found

The ranking, with every picture, is in our test of the best AI image generator, and the two OpenAI variants meet in GPT Image 2.5 Sunburst vs Flare. No person has checked the judges yet; the ranking says so, and treats a gap of one or two checks between models as noise. Pooled across the eight models:

Every model passed every check on binding, counting, photoreal and style. The differences sit in anatomy, exclusion, text and layout. Ignoring spaces in OCR moved text from 93% to 98% and layout from 94% to 99%.

Build your own image generation benchmark: a checklist

1. Write prompts whose facts can be checked, one yes/no question per fact, each tagged with the ability it tests.
2. Centre the prompts on people: faces, hands, two or more people with attributes to bind, the same person across several shots. That is what readers care about, and it is where our next round goes; our two-person and portrait prompts were passed by every model, so the next ones must be harder.
3. Word each question so it reads only one way, and let a second judge find the ones that do not.
4. Read text twice: OCR with spaces and punctuation ignored, a judge for exact spelling and layout, and a rule decided in advance for when they differ.
5. Blind the pictures: shuffle and re-letter them per prompt with a fixed seed; keep the key out of reach.
6. Use two judges that answer apart, a third for disagreements, and report agreement and kappa together.
7. Calibrate against people before you base decisions on a ranking, per kind of check. We have not run that round yet.
8. Cap the spending in the script, and on the API key.
9. Make several pictures per prompt if the budget allows, so a pass is a rate.
10. Publish the prompts, checks and pictures, so anyone can check a verdict by looking. Ours: the spec, the judges' rules, and all 96 pictures in the ranking.

Why we run tests ourselves instead of copying makers' numbers is in where LLM leaderboard scores come from; we graded our own films blind the same way when we compared language models (the cheaper model wasn't half the price).

How we tested, and where the numbers come from

Every number above comes from one run on 25 October 2026. The limits: one picture per prompt; twelve prompts, too few to separate close models; judges checked against each other, not yet against people; and models our API account could not reach (FLUX 3, Qwen-Image 2.1, Recraft V4.1 and Krea among them) are missing. Disagree with a verdict? Write to [email protected] and we will look again.

FAQ

What is LLM-as-a-judge for images?

It means asking a vision-language model to evaluate a generated picture, usually by answering questions about it; it is also called VLM as a judge. The most reliable form asks one yes/no question per fact in the prompt instead of an overall score: in GenEval 2, a scorer built that way matched people's labels with an AUROC of 94.5%.

How accurate is an AI judge compared with people?

On simple facts, close: in GenEval 2 the best checklist scorers rank a picture people marked right above one they marked wrong about 92 to 95% of the time. Our two Claude judges agreed with each other on 95.7% of answers, which measures consistency, not accuracy; check a sample against people first.

What is GenEval?

GenEval is a 2023 text-to-image benchmark that checks objects, counts, colours and positions with an object detector. GenEval 2 (December 2025) has a vision-language model answer one question per fact instead, and found the original had drifted from human judgement by up to 17.7 points.

Should I use OCR or a vision model to check text in AI images?

Both. OCR is free and repeatable, but in our run 10 of its 11 failures were a dropped space, so we now ignore spaces; after that, OCR and the judges agreed on 22 of 24 text facts. A vision model reads spacing, punctuation and layout, but two copies of it can share blind spots. Where the two disagree, look at the picture.

How much does it cost to judge AI images with Claude?

A 1024-pixel picture is 1,369 input tokens, about half a cent at Claude Opus 5.5's $4 per million. With the questions, answers and thinking, two judges reading our 96 pictures would cost about $2.89 through the API, or $1.44 with the Batch API.

Make an explainer