Is the failure class right?

A small language model comments on failed test jobs of openSUSE. I tried to make its verdicts better. The hard part was telling whether they got better.

Contents

On openqa.opensuse.org, a small language model comments on failed test jobs. It gets what a human reviewer would look at and says what kind of failure it sees. In the autumn of 2026 I tuned it. The first pull requests are from 25 October, and most of the work was done in the week of 7 to 14 November.

I started mostly out of curiosity, to see how much I could improve with constrained resources. The constraint was the small model. I had no particular goal beyond exploration and perhaps some improvement.

The short version:

  • Small fixes to the prompt and the script made visible differences.
  • Whether the model’s verdict is right could hardly be measured, because too few failures had a cause recorded by a person.
  • One change looked like a clear win until I checked where the right answers it was scored against came from.
  • What looked most useful was a plain rerun. The comment already said whether to restart a job, and no code acted on that line.

What openQA does and what the model adds

openQA tests an operating system the way a person would. It boots a machine, types, clicks, and compares the screen with reference screenshots, called needles, and the serial console with expected text. One run of one test scenario on one build of the system is a job. openqa.opensuse.org, o3 for short, is the public instance that tests openSUSE.

When a job fails, somebody has to decide what kind of failure it is. There are three kinds:

  • a product bug: the software under test is wrong;
  • a test issue: the test is wrong or out of date;
  • infrastructure: a worker (the machine that runs the job), the network or some outside server let the job down.

A reviewer records the decision by linking a bug or a ticket on the job. That link is called a bug reference.

o3 also asks a language model. When a job fails and no known issue matches it, a hook on o3 runs a script called openqa-llm-investigate. The script sends what a reviewer would look at to Gemma 4 E4B. The model runs under llama.cpp on three of the worker hosts, without a graphics card, in a heavily compressed form (quantised to a level called IQ2_M). Its reply becomes a comment on the job:

**LLM Investigation summary:**

* boot_to_desktop: The test failed because no candidate needle matching the expected desktop environment or display manager was found.

The `gnuhealth` test failed in the `boot_to_desktop` module because the system failed to find a matching screen (`needle`) that contained tags such as `generic-desktop` or `displaymanager`. This indicates that the expected graphical environment did not load successfully during the boot process.

Classification: Product bug BISECT: NO — 18 other job(s) in this build fail in the same module Restart: NO — the needle does not match the screen

This comment has been generated by [openqa-llm-investigate](https://github.com/os-autoinst/os-autoinst-scripts/blob/master/openqa-llm-investigate). flag:verbatim

A comment as the model posted it on 4 November 2026, shown as its Markdown source.

The three lines before the footer are meant for programs: the class, whether to bisect, and whether to restart the job. Only BISECT: YES starts anything: the hook then schedules investigation jobs that retry the test and search for the change that broke it.

From a failed job to a comment The model gives three answers per job, and only one of them is wired to anything. Test job fails the job-done hook finds no known issue and runs openqa-llm-investigate One prompt failure analysis of the log earlier reviews of the module settings diff to last good job test commits since then llama.cpp server Gemma 4 E4B, 3 hosts, nginx Q1 analysis per module Q2 summary, at most 2 sentences Q3 verdict, with job statistics the script posts the replies Comment on the job ends with three lines: Classification: Product bug | Test issue | Infrastructure problem People only reviewed 2.2% of commented jobs read BISECT: YES | NO openqa-investigate retry, last good tests and build, then openqa-trigger-bisect-jobs YES Restart: YES | NO not connected: nothing acts on it
From a failed job to a comment The model gives three answers per job, and only one of them is wired to anything. Test job fails the job-done hook finds no known issue and runs openqa-llm-investigate One prompt failure analysis of the log earlier reviews of the module settings diff to last good job test commits since then llama.cpp server Gemma 4 E4B, 3 hosts, nginx Q1 analysis per module Q2 summary, at most 2 sentences Q3 verdict, with job statistics the script posts the replies Job comment ends with three lines: Classification: Product bug | Test issue | Infrastructure problem People only reviewed 2.2% of commented jobs BISECT: YES | NO openqa-investigate retry, last good tests and build, then bisect jobs YES Restart: YES | NO not connected: nothing acts on it
Figure 1. How a failed job gets its comment. Of the three closing lines, people read the class, the hook acts on BISECT: YES, and nothing is connected to Restart.
Data
  1. A test job fails on openQA. If no known issue matches, the job-done hook runs openqa-llm-investigate.
  2. The script builds one prompt from:
    1. the failure analysis of the job's log, with earlier reviews of runs failing in the same module
    2. the settings difference to the last good job
    3. the test commits since the last good job
  3. It asks a llama.cpp server running Gemma 4 E4B (three hosts behind nginx) three questions in one chat:
    1. question 1: the analysis, one line per failed module
    2. question 2: a summary of at most two sentences
    3. question 3: the verdict, with the job statistics
  4. The script posts a comment on the job that ends with three lines:
    1. Classification: Product bug | Test issue | Infrastructure problem. Only people read it; they reviewed 2.2% of the commented jobs.
    2. BISECT: YES | NO. On YES the hook starts openqa-investigate (a retry and reruns with the last good tests and build) and then openqa-trigger-bisect-jobs. This is the only answer wired to anything.
    3. Restart: YES | NO. Not connected: nothing acts on it.

In 42 days the model commented on 6459 jobs in four job groups. A person later reviewed about 140 of them, 2.2%.

The easy fixes

The first fixes needed no statistics.

The prompt, the text that tells the model what to do, carried a hint about outages that fired on nearly every build. On 17 jobs I audited by hand, the model called 15 an infrastructure problem. After the prompt required evidence for that verdict, 3 of 17 were called infrastructure, and the class matched my audit on 12 of 17 jobs, up from 4 (#802).

Half of the commented jobs, 3290 or 51%, were in the Development group, where a person reviewed 4. The script now skips that group (#810).

The third fix is about length. A model reads text as tokens, which are pieces of words, and can take in only so many. The script can cut a long prompt to fit such a limit, if one is set; as far as I can see, o3 sets none. It cuts by characters and assumed 2.5 characters per token (#799). Most real prompts have fewer, so a cut prompt can still be too long. I proposed 1.9 (#815).

Characters per token in 520 measured prompts Real prompts hold a median of 2.28 characters per token, fewer than the 2.5 the script assumed, so prompts cut by character count still overflowed. 0 20 40 60 80 100 1.8 2.0 2.2 2.4 2.6 2.8 3.0 characters per token 1.70–1.75 characters per token: 1 prompt 1.75–1.80 characters per token: 0 prompts 1.80–1.85 characters per token: 0 prompts 1.85–1.90 characters per token: 0 prompts 1.90–1.95 characters per token: 1 prompt 1.95–2.00 characters per token: 3 prompts 2.00–2.05 characters per token: 18 prompts 2.05–2.10 characters per token: 11 prompts 2.10–2.15 characters per token: 29 prompts 2.15–2.20 characters per token: 65 prompts 2.20–2.25 characters per token: 79 prompts 2.25–2.30 characters per token: 92 prompts 2.30–2.35 characters per token: 57 prompts 2.35–2.40 characters per token: 36 prompts 2.40–2.45 characters per token: 23 prompts 2.45–2.50 characters per token: 19 prompts 2.50–2.55 characters per token: 23 prompts 2.55–2.60 characters per token: 15 prompts 2.60–2.65 characters per token: 5 prompts 2.65–2.70 characters per token: 4 prompts 2.70–2.75 characters per token: 4 prompts 2.75–2.80 characters per token: 8 prompts 2.80–2.85 characters per token: 1 prompt 2.85–2.90 characters per token: 2 prompts 2.90–2.95 characters per token: 0 prompts 2.95–3.00 characters per token: 0 prompts 3.00–3.05 characters per token: 0 prompts 3.05–3.10 characters per token: 1 prompt 3.10 and more (3.10–3.95, 17 bins drawn as one bar): 23 prompts prompts assumed 2.5 median 2.28 proposed 1.9 3.1 and more 83% of prompts are below 2.5
Characters per token in 520 measured prompts Real prompts hold a median of 2.28 characters per token, fewer than the 2.5 the script assumed, so prompts cut by character count still overflowed. 0 25 50 75 100 2.0 2.5 3.0 characters per token 1.70–1.75 characters per token: 1 prompt 1.75–1.80 characters per token: 0 prompts 1.80–1.85 characters per token: 0 prompts 1.85–1.90 characters per token: 0 prompts 1.90–1.95 characters per token: 1 prompt 1.95–2.00 characters per token: 3 prompts 2.00–2.05 characters per token: 18 prompts 2.05–2.10 characters per token: 11 prompts 2.10–2.15 characters per token: 29 prompts 2.15–2.20 characters per token: 65 prompts 2.20–2.25 characters per token: 79 prompts 2.25–2.30 characters per token: 92 prompts 2.30–2.35 characters per token: 57 prompts 2.35–2.40 characters per token: 36 prompts 2.40–2.45 characters per token: 23 prompts 2.45–2.50 characters per token: 19 prompts 2.50–2.55 characters per token: 23 prompts 2.55–2.60 characters per token: 15 prompts 2.60–2.65 characters per token: 5 prompts 2.65–2.70 characters per token: 4 prompts 2.70–2.75 characters per token: 4 prompts 2.75–2.80 characters per token: 8 prompts 2.80–2.85 characters per token: 1 prompt 2.85–2.90 characters per token: 2 prompts 2.90–2.95 characters per token: 0 prompts 2.95–3.00 characters per token: 0 prompts 3.00–3.05 characters per token: 0 prompts 3.05–3.10 characters per token: 1 prompt 3.10 and more (3.10–3.95, 17 bins drawn as one bar): 23 prompts prompts assumed 2.5 median 2.28 proposed 1.9 3.1 and more 83% of prompts are below 2.5
Figure 2. Each bar counts how many of 520 real prompts, 122 of them from an internal instance, have that many characters per token. The median is 2.28 and 83% of the prompts are below the assumed 2.5, so they hold more tokens than the script expects.
Data
Characters per token in 520 prompts, in bins of 0.05: median 2.28, 83% below the assumed 2.5 (434 prompts), 1 prompt below the proposed 1.9. The figure draws the 17 bins from 3.10 to 3.95 as one bar of 23 prompts, labelled 3.1 and more.
Characters per tokenPrompts
1.70–1.751
1.75–1.800
1.80–1.850
1.85–1.900
1.90–1.951
1.95–2.003
2.00–2.0518
2.05–2.1011
2.10–2.1529
2.15–2.2065
2.20–2.2579
2.25–2.3092
2.30–2.3557
2.35–2.4036
2.40–2.4523
2.45–2.5019
2.50–2.5523
2.55–2.6015
2.60–2.655
2.65–2.704
2.70–2.754
2.75–2.808
2.80–2.851
2.85–2.902
2.90–2.950
2.95–3.000
3.00–3.050
3.05–3.101
3.10–3.151
3.15–3.201
3.20–3.251
3.25–3.301
3.30–3.352
3.35–3.400
3.40–3.453
3.45–3.501
3.50–3.552
3.55–3.604
3.60–3.651
3.65–3.702
3.70–3.751
3.75–3.801
3.80–3.850
3.85–3.901
3.90–3.951
All prompts520

Too few labels

To say that a change makes the class right more often, I need failures whose class is known. Such a known class is a label.

Reviewers are the natural source. In 42 days, 1316 failed jobs carried a bug reference, but 1226 of them got it from a bot or by carry-over from an earlier job of the same scenario. 90 were set by a person, and of the 83 whose ticket is public, not one is an infrastructure problem. A network hiccup usually gets a restart, not a ticket.

Going back further gave 98 jobs with a reference set by a person, 50 product bugs and 48 test issues. This is the gold set. It comes from 27 tickets, since one bug can fail many jobs.

For infrastructure I had a frontier model, one of the most capable models available, label 299 failed o3 jobs. This is the silver set. No person checked these labels, and the labeller had hindsight, which the commenting model lacks: it could see whether the same scenario passed an hour later.

Where the class labels come from: people, the gold set and the silver set People labelled 90 of 1,316 failed jobs with a bug reference, and none of the 83 with a public ticket is infrastructure; the 98 gold jobs come from 27 tickets, one of them 18 jobs; all 111 infrastructure labels come from the 299-job silver set labelled by a frontier model with hindsight. product bug test issue infrastructure class unknown unclear or another team ticket that does not map cleanly 90 of 1,316 failed jobs with a bug reference in 42 days were labelled by a person Set by a bot or carried over: 1,226 jobs 1,226 set by a bot or carried over Set by a person: 90 jobs Set by a person, product bug: 41 jobs 41 Set by a person, test issue: 40 jobs 40 Set by a person, ticket that is not public, class unknown: 7 jobs Set by a person, ticket of another team: 2 jobs None of the 83 with a public ticket is infrastructure. 7 point to a ticket that is not public, so their class is unknown; 2 point to another team's ticket. 98 jobs in the gold set, from 27 tickets Ticket 1 of 27: 18 jobs, test issue, does not map cleanly onto the three classes 18 Ticket 2 of 27: 8 jobs, product bug 8 Ticket 3 of 27: 8 jobs, test issue 8 Ticket 4 of 27: 7 jobs, product bug 7 Ticket 5 of 27: 7 jobs, product bug, does not map cleanly onto the three classes Ticket 6 of 27: 6 jobs, product bug Ticket 7 of 27: 5 jobs, test issue, does not map cleanly onto the three classes Ticket 8 of 27: 4 jobs, product bug Ticket 9 of 27: 4 jobs, product bug Ticket 10 of 27: 4 jobs, test issue, does not map cleanly onto the three classes Ticket 11 of 27: 3 jobs, product bug Ticket 12 of 27: 3 jobs, test issue Ticket 13 of 27: 2 jobs, product bug Ticket 14 of 27: 2 jobs, product bug Ticket 15 of 27: 2 jobs, product bug Ticket 16 of 27: 2 jobs, product bug, does not map cleanly onto the three classes Ticket 17 of 27: 2 jobs, test issue Ticket 18 of 27: 2 jobs, test issue, does not map cleanly onto the three classes Ticket 19 of 27: 1 job, product bug Ticket 20 of 27: 1 job, product bug Ticket 21 of 27: 1 job, product bug Ticket 22 of 27: 1 job, test issue Ticket 23 of 27: 1 job, test issue Ticket 24 of 27: 1 job, test issue Ticket 25 of 27: 1 job, test issue Ticket 26 of 27: 1 job, test issue, does not map cleanly onto the three classes Ticket 27 of 27: 1 job, test issue, does not map cleanly onto the three classes No ticket is labelled infrastructure. 8 of the 27 tickets, with 40 of the 98 jobs, do not map cleanly onto the three classes, including the largest (18 jobs). 299 jobs in the silver set, labelled by a frontier model with hindsight Silver set, product bug: 87 jobs, 70 of them confident (labeller confidence at least 0.6) 87 Silver set, test issue: 87 jobs, 26 of them confident (labeller confidence at least 0.6) 87 Silver set, infrastructure: 111 jobs, 57 of them confident (labeller confidence at least 0.6) 111 Silver set, unclear: 14 jobs The only infrastructure labels: 111 jobs. 14 stay unclear.
Where the class labels come from: people, the gold set and the silver set People labelled 90 of 1,316 failed jobs with a bug reference, and none of the 83 with a public ticket is infrastructure; the 98 gold jobs come from 27 tickets, one of them 18 jobs; all 111 infrastructure labels come from the 299-job silver set labelled by a frontier model with hindsight. product bug test issue infrastructure class unknown unclear or another team ticket that does not map cleanly 90 of 1,316 failed jobs with a bug reference in 42 days were labelled by a person Set by a bot or carried over: 1,226 jobs 1,226 set by a bot or carried over Set by a person: 90 jobs Set by a person, product bug: 41 jobs 41 Set by a person, test issue: 40 jobs 40 Set by a person, ticket that is not public, class unknown: 7 jobs Set by a person, ticket of another team: 2 jobs None of the 83 with a public ticket is infrastructure. 7 point to a ticket that is not public, so their class is unknown; 2 point to another team's ticket. 98 jobs in the gold set, from 27 tickets Ticket 1 of 27: 18 jobs, test issue, does not map cleanly onto the three classes 18 Ticket 2 of 27: 8 jobs, product bug 8 Ticket 3 of 27: 8 jobs, test issue 8 Ticket 4 of 27: 7 jobs, product bug 7 Ticket 5 of 27: 7 jobs, product bug, does not map cleanly onto the three classes Ticket 6 of 27: 6 jobs, product bug Ticket 7 of 27: 5 jobs, test issue, does not map cleanly onto the three classes Ticket 8 of 27: 4 jobs, product bug Ticket 9 of 27: 4 jobs, product bug Ticket 10 of 27: 4 jobs, test issue, does not map cleanly onto the three classes Ticket 11 of 27: 3 jobs, product bug Ticket 12 of 27: 3 jobs, test issue Ticket 13 of 27: 2 jobs, product bug Ticket 14 of 27: 2 jobs, product bug Ticket 15 of 27: 2 jobs, product bug Ticket 16 of 27: 2 jobs, product bug, does not map cleanly onto the three classes Ticket 17 of 27: 2 jobs, test issue Ticket 18 of 27: 2 jobs, test issue, does not map cleanly onto the three classes Ticket 19 of 27: 1 job, product bug Ticket 20 of 27: 1 job, product bug Ticket 21 of 27: 1 job, product bug Ticket 22 of 27: 1 job, test issue Ticket 23 of 27: 1 job, test issue Ticket 24 of 27: 1 job, test issue Ticket 25 of 27: 1 job, test issue Ticket 26 of 27: 1 job, test issue, does not map cleanly onto the three classes Ticket 27 of 27: 1 job, test issue, does not map cleanly onto the three classes No ticket is labelled infrastructure. 8 of the 27 tickets, with 40 of the 98 jobs, do not map cleanly onto the three classes, including the largest (18 jobs). 299 jobs in the silver set, labelled by a frontier model with hindsight Silver set, product bug: 87 jobs, 70 of them confident (labeller confidence at least 0.6) 87 Silver set, test issue: 87 jobs, 26 of them confident (labeller confidence at least 0.6) 87 Silver set, infrastructure: 111 jobs, 57 of them confident (labeller confidence at least 0.6) 111 Silver set, unclear: 14 jobs The only infrastructure labels: 111 jobs. 14 stay unclear.
Figure 3. Where the labels come from. Top: of 1316 failed jobs with a bug reference, a person set the 90 at the right end, enlarged below. Middle: the 98 gold jobs, one square per job and one group per ticket; hollow squares are jobs on tickets that do not map cleanly onto a class. Bottom: the 299 jobs a model labelled. Green, infrastructure, appears only there.
Data
Where the class labels come from: bug references set by people, the gold set by ticket (largest first) and the silver set labelled by a frontier model with hindsight
SourcePartJobsNote
Bug referencesfailed jobs with a bug reference1,31642 days
Bug referencesset by a bot or carried over1,226
Bug referencesset by a person9023 distinct references
Set by a personproduct bug41
Set by a persontest issue40
Set by a personclass unknown7ticket that is not public
Set by a personticket of another team2
Set by a personinfrastructure0none of the 83 with a public ticket
Gold setall jobs9827 tickets
Gold setproduct bug5014 tickets
Gold settest issue4813 tickets
Gold setdo not map cleanly onto the three classes408 tickets
Gold setticket 118test issue, does not map cleanly
Gold setticket 28product bug
Gold setticket 38test issue
Gold setticket 47product bug
Gold setticket 57product bug, does not map cleanly
Gold setticket 66product bug
Gold setticket 75test issue, does not map cleanly
Gold setticket 84product bug
Gold setticket 94product bug
Gold setticket 104test issue, does not map cleanly
Gold setticket 113product bug
Gold setticket 123test issue
Gold setticket 132product bug
Gold setticket 142product bug
Gold setticket 152product bug
Gold setticket 162product bug, does not map cleanly
Gold setticket 172test issue
Gold setticket 182test issue, does not map cleanly
Gold setticket 191product bug
Gold setticket 201product bug
Gold setticket 211product bug
Gold setticket 221test issue
Gold setticket 231test issue
Gold setticket 241test issue
Gold setticket 251test issue
Gold setticket 261test issue, does not map cleanly
Gold setticket 271test issue, does not map cleanly
Silver setall jobs299labelled by a frontier model with hindsight
Silver setproduct bug8770 confident (labeller confidence at least 0.6)
Silver settest issue8726 confident (labeller confidence at least 0.6)
Silver setinfrastructure11157 confident (labeller confidence at least 0.6)
Silver setunclear14

Jobs that share a ticket are not independent, so what counts is the number of tickets. With 27, a comparison of two models can reliably show only a difference of about 30 percentage points. Fifteen points would need 105 tickets, and ten points 235.

Smallest detectable difference against the number of labelled tickets With 27 labelled tickets a paired comparison can show a difference of about 30 points and nothing smaller; 15 points would need 105 tickets, 10 points 235. 0 10 20 30 40 50 smallest detectable difference, points 0 50 100 150 200 250 300 labelled tickets 27 tickets: what I had 30 points 27 tickets: smallest difference that shows is about 30 points (what I had) 105 tickets 15 points 105 tickets: smallest difference that shows is about 15 points 235 tickets 10 points 235 tickets: smallest difference that shows is about 10 points too small to show with that many tickets
Smallest detectable difference against the number of labelled tickets With 27 labelled tickets a paired comparison can show a difference of about 30 points and nothing smaller; 15 points would need 105 tickets, 10 points 235. 0 10 20 30 40 50 smallest detectable difference, points 0 100 200 300 labelled tickets 27 tickets: what I had 30 points 27 tickets: smallest difference that shows is about 30 points (what I had) 105 tickets 15 points 105 tickets: smallest difference that shows is about 15 points 235 tickets 10 points 235 tickets: smallest difference that shows is about 10 points too small to show with that many tickets
Figure 4. The smallest difference between two models that a labelled set of a given size can show, for a paired comparison at the usual 5% level and 80% power, if about a third of the tickets change between the two. Anything in the grey area is too small to show reliably.
Data
Smallest difference in percentage points that a paired comparison can show, by number of labelled tickets: 2.8 for the confidence wanted, times 0.55 for how much paired answers vary per ticket, divided by the square root of the tickets; marked rows are the points drawn in the figure, with the rounded value shown there
TicketsSmallest detectable difference (points)Marked
1048.7
1539.8
2034.4
2530.8
2729.6what I had, about 30
3028.1
3526.0
4024.3
4523.0
5021.8
5520.8
6019.9
6519.1
7018.4
7517.8
8017.2
8516.7
9016.2
9515.8
10015.4
10515.0about 15
11014.7
12014.1
13013.5
14013.0
15012.6
16012.2
17011.8
18011.5
19011.2
20010.9
21010.6
22010.4
23010.2
23510.0about 10
2409.9
2509.7
2609.6
2709.4
2809.2
2909.0
3008.9

A noisy model

Asked the same thing twice, the model does not always give the same class. For most experiments I used Q8, a less compressed version of the same Gemma. A second run a day later, with the same prompt and the same settings, kept the class on 29 of 35 gold jobs.

So if a new prompt gets three more jobs right than the old one, one run each, it has shown nothing.

Jobs that kept their class when the same prompt ran twice The same prompt run twice kept the class on 29 of 35 gold jobs, 11 of 16 test issues among them and 7 of 16 gold jobs run again four days later, half of them from that set, against 33 of 34 product bugs from an internal instance. kept its class changed Gold jobs, run again a day later: 29 of 35 kept the class Gold jobs, run again a day later: 6 of 35 changed Test issues among those gold jobs: 11 of 16 kept the class Test issues among those gold jobs: 5 of 16 changed Another 16 gold jobs (8 shared), run again four days later: 7 of 16 kept the class Another 16 gold jobs (8 shared), run again four days later: 9 of 16 changed Product bugs from an internal instance (internal counts): 33 of 34 kept the class Product bugs from an internal instance (internal counts): 1 of 34 changed Gold jobs run again a day later 29 of 35 kept Test issues among them 11 of 16 kept Another 16 gold jobs (8 shared) run again four days later 7 of 16 kept Product bugs, internal instance internal counts 33 of 34 kept
Jobs that kept their class when the same prompt ran twice The same prompt run twice kept the class on 29 of 35 gold jobs, 11 of 16 test issues among them and 7 of 16 gold jobs run again four days later, half of them from that set, against 33 of 34 product bugs from an internal instance. kept its class changed Gold jobs, run again a day later: 29 of 35 kept the class Gold jobs, run again a day later: 6 of 35 changed Test issues among those gold jobs: 11 of 16 kept the class Test issues among those gold jobs: 5 of 16 changed Another 16 gold jobs (8 shared), run again four days later: 7 of 16 kept the class Another 16 gold jobs (8 shared), run again four days later: 9 of 16 changed Product bugs from an internal instance (internal counts): 33 of 34 kept the class Product bugs from an internal instance (internal counts): 1 of 34 changed 29 of 35 kept Gold jobs run again a day later 11 of 16 kept Test issues among them 7 of 16 kept Another 16 gold jobs (8 shared) run again four days later 33 of 34 kept Product bugs, internal instance internal counts
Figure 5. Each bar is a set of jobs run twice with the same prompt and the same settings. Grey: the class stayed the same. Purple: it changed. In one set of 16 jobs, run again after four days, more than half changed.
Data
Jobs that kept their class when the same prompt ran twice (Gemma 4 E4B Q8_0, same prompt, same settings)
Job setJobsKept its classChanged
Gold jobs, run again a day later35296
Test issues among those gold jobs16115
Another 16 gold jobs (8 shared), run again four days later1679
Product bugs from an internal instance (internal counts)34331
Gold jobs and internal product bugs together (mixed counts; context, not drawn)69627
Verdict only, asked again on the same stored answers to the first two questions (mixed counts; context, not drawn)36333

What the model gets wrong

Gemma leans towards product bug. Of the 46 test-issue jobs in the gold set that it answered, Gemma at Q8 called 39 a product bug and 6 a test issue. The IQ2_M version that runs in production called 15 of the 46 a test issue, but called only 35 of the 49 product bugs a product bug. Both got 50 of 95 jobs right.

What Gemma at Q8 answered against the label a person set, by kind of failure Of the jobs a person labelled test issue, Gemma at Q8 called 39 product bugs and 6 test issues; the IQ2_M quantisation in production called 15 test issues. Answered product bug test issue infrastructure Gemma at Q8, labels set by people IQ2_M, the quantisation in production, on the same jobs No job a person labelled is infrastructure. Silver jobs labelled infrastructure afterwards, against the class posted in production product bug 49 jobs Gemma at Q8, all jobs: labelled product bug, answered product bug: 44 of 49 jobs (90%) 44 Gemma at Q8, all jobs: labelled product bug, answered test issue: 5 of 49 jobs (10%) 5 44 of 49 called product bug test issue 46 jobs Gemma at Q8, all jobs: labelled test issue, answered product bug: 39 of 46 jobs (85%) 39 Gemma at Q8, all jobs: labelled test issue, answered test issue: 6 of 46 jobs (13%) 6 Gemma at Q8, all jobs: labelled test issue, answered infrastructure: 1 of 46 jobs (2%) 39 of 46 called product bug, 6 test issue product bug 49 jobs IQ2_M, all jobs: labelled product bug, answered product bug: 35 of 49 jobs (71%) 35 IQ2_M, all jobs: labelled product bug, answered test issue: 13 of 49 jobs (27%) 13 IQ2_M, all jobs: labelled product bug, answered infrastructure: 1 of 49 jobs (2%) test issue 46 jobs IQ2_M, all jobs: labelled test issue, answered product bug: 31 of 46 jobs (67%) 31 IQ2_M, all jobs: labelled test issue, answered test issue: 15 of 46 jobs (33%) 15 15 of 46 called test issue infrastructure 111 jobs Silver, posted, all jobs: labelled infrastructure, posted product bug: 33 of 111 jobs (30%) 33 Silver, posted, all jobs: labelled infrastructure, posted test issue: 45 of 111 jobs (41%) 45 Silver, posted, all jobs: labelled infrastructure, posted infrastructure: 33 of 111 jobs (30%) 33 product bug 22 jobs Gemma at Q8, failed in a script: labelled product bug, answered product bug: 22 of 22 jobs (100%) 22 22 of 22 called product bug test issue 13 jobs Gemma at Q8, failed in a script: labelled test issue, answered product bug: 13 of 13 jobs (100%) 13 13 of 13 called product bug, none a test issue product bug 22 jobs IQ2_M, failed in a script: labelled product bug, answered product bug: 20 of 22 jobs (91%) 20 IQ2_M, failed in a script: labelled product bug, answered test issue: 2 of 22 jobs (9%) 2 test issue 13 jobs IQ2_M, failed in a script: labelled test issue, answered product bug: 8 of 13 jobs (62%) 8 IQ2_M, failed in a script: labelled test issue, answered test issue: 5 of 13 jobs (38%) 5 5 of 13 called test issue infrastructure 53 jobs Silver, posted, failed in a script: labelled infrastructure, posted product bug: 14 of 53 jobs (26%) 14 Silver, posted, failed in a script: labelled infrastructure, posted test issue: 17 of 53 jobs (32%) 17 Silver, posted, failed in a script: labelled infrastructure, posted infrastructure: 22 of 53 jobs (42%) 22 product bug 20 jobs Gemma at Q8, needle mismatch: labelled product bug, answered product bug: 15 of 20 jobs (75%) 15 Gemma at Q8, needle mismatch: labelled product bug, answered test issue: 5 of 20 jobs (25%) 5 15 of 20 called product bug test issue 24 jobs Gemma at Q8, needle mismatch: labelled test issue, answered product bug: 18 of 24 jobs (75%) 18 Gemma at Q8, needle mismatch: labelled test issue, answered test issue: 6 of 24 jobs (25%) 6 18 of 24 called product bug, 6 test issue product bug 20 jobs IQ2_M, needle mismatch: labelled product bug, answered product bug: 9 of 20 jobs (45%) 9 IQ2_M, needle mismatch: labelled product bug, answered test issue: 11 of 20 jobs (55%) 11 test issue 24 jobs IQ2_M, needle mismatch: labelled test issue, answered product bug: 15 of 24 jobs (62%) 15 IQ2_M, needle mismatch: labelled test issue, answered test issue: 9 of 24 jobs (38%) 9 9 of 24 called test issue infrastructure 29 jobs Silver, posted, needle mismatch: labelled infrastructure, posted product bug: 7 of 29 jobs (24%) 7 Silver, posted, needle mismatch: labelled infrastructure, posted test issue: 20 of 29 jobs (69%) 20 Silver, posted, needle mismatch: labelled infrastructure, posted infrastructure: 2 of 29 jobs (7%) 2 product bug 3 jobs Gemma at Q8, serial console: labelled product bug, answered product bug: 3 of 3 jobs (100%) 3 3 of 3 called product bug test issue 7 jobs Gemma at Q8, serial console: labelled test issue, answered product bug: 6 of 7 jobs (86%) 6 Gemma at Q8, serial console: labelled test issue, answered infrastructure: 1 of 7 jobs (14%) 1 6 of 7 called product bug, none a test issue product bug 3 jobs IQ2_M, serial console: labelled product bug, answered product bug: 2 of 3 jobs (67%) 2 IQ2_M, serial console: labelled product bug, answered infrastructure: 1 of 3 jobs (33%) 1 test issue 7 jobs IQ2_M, serial console: labelled test issue, answered product bug: 6 of 7 jobs (86%) 6 IQ2_M, serial console: labelled test issue, answered test issue: 1 of 7 jobs (14%) 1 1 of 7 called test issue The silver set has no serial console split.
What Gemma at Q8 answered against the label a person set, by kind of failure Of the jobs a person labelled test issue, Gemma at Q8 called 39 product bugs and 6 test issues; the IQ2_M quantisation in production called 15 test issues. Answered product bug test issue infrastructure Gemma at Q8, labels set by people IQ2_M, the quantisation in production, on the same jobs No job a person labelled is infrastructure. Silver jobs labelled infrastructure afterwards, against the class posted in production product bug 49 jobs Gemma at Q8, all jobs: labelled product bug, answered product bug: 44 of 49 jobs (90%) 44 Gemma at Q8, all jobs: labelled product bug, answered test issue: 5 of 49 jobs (10%) 5 44 of 49 called product bug test issue 46 jobs Gemma at Q8, all jobs: labelled test issue, answered product bug: 39 of 46 jobs (85%) 39 Gemma at Q8, all jobs: labelled test issue, answered test issue: 6 of 46 jobs (13%) 6 Gemma at Q8, all jobs: labelled test issue, answered infrastructure: 1 of 46 jobs (2%) 39 of 46 called product bug, 6 test issue product bug 49 jobs IQ2_M, all jobs: labelled product bug, answered product bug: 35 of 49 jobs (71%) 35 IQ2_M, all jobs: labelled product bug, answered test issue: 13 of 49 jobs (27%) 13 IQ2_M, all jobs: labelled product bug, answered infrastructure: 1 of 49 jobs (2%) test issue 46 jobs IQ2_M, all jobs: labelled test issue, answered product bug: 31 of 46 jobs (67%) 31 IQ2_M, all jobs: labelled test issue, answered test issue: 15 of 46 jobs (33%) 15 15 of 46 called test issue infrastructure 111 jobs Silver, posted, all jobs: labelled infrastructure, posted product bug: 33 of 111 jobs (30%) 33 Silver, posted, all jobs: labelled infrastructure, posted test issue: 45 of 111 jobs (41%) 45 Silver, posted, all jobs: labelled infrastructure, posted infrastructure: 33 of 111 jobs (30%) 33 product bug 22 jobs Gemma at Q8, failed in a script: labelled product bug, answered product bug: 22 of 22 jobs (100%) 22 22 of 22 called product bug test issue 13 jobs Gemma at Q8, failed in a script: labelled test issue, answered product bug: 13 of 13 jobs (100%) 13 13 of 13 called product bug, none a test issue product bug 22 jobs IQ2_M, failed in a script: labelled product bug, answered product bug: 20 of 22 jobs (91%) 20 IQ2_M, failed in a script: labelled product bug, answered test issue: 2 of 22 jobs (9%) 2 test issue 13 jobs IQ2_M, failed in a script: labelled test issue, answered product bug: 8 of 13 jobs (62%) 8 IQ2_M, failed in a script: labelled test issue, answered test issue: 5 of 13 jobs (38%) 5 5 of 13 called test issue infrastructure 53 jobs Silver, posted, failed in a script: labelled infrastructure, posted product bug: 14 of 53 jobs (26%) 14 Silver, posted, failed in a script: labelled infrastructure, posted test issue: 17 of 53 jobs (32%) 17 Silver, posted, failed in a script: labelled infrastructure, posted infrastructure: 22 of 53 jobs (42%) 22 product bug 20 jobs Gemma at Q8, needle mismatch: labelled product bug, answered product bug: 15 of 20 jobs (75%) 15 Gemma at Q8, needle mismatch: labelled product bug, answered test issue: 5 of 20 jobs (25%) 5 15 of 20 called product bug test issue 24 jobs Gemma at Q8, needle mismatch: labelled test issue, answered product bug: 18 of 24 jobs (75%) 18 Gemma at Q8, needle mismatch: labelled test issue, answered test issue: 6 of 24 jobs (25%) 6 18 of 24 called product bug, 6 test issue product bug 20 jobs IQ2_M, needle mismatch: labelled product bug, answered product bug: 9 of 20 jobs (45%) 9 IQ2_M, needle mismatch: labelled product bug, answered test issue: 11 of 20 jobs (55%) 11 test issue 24 jobs IQ2_M, needle mismatch: labelled test issue, answered product bug: 15 of 24 jobs (62%) 15 IQ2_M, needle mismatch: labelled test issue, answered test issue: 9 of 24 jobs (38%) 9 9 of 24 called test issue infrastructure 29 jobs Silver, posted, needle mismatch: labelled infrastructure, posted product bug: 7 of 29 jobs (24%) 7 Silver, posted, needle mismatch: labelled infrastructure, posted test issue: 20 of 29 jobs (69%) 20 Silver, posted, needle mismatch: labelled infrastructure, posted infrastructure: 2 of 29 jobs (7%) 2 product bug 3 jobs Gemma at Q8, serial console: labelled product bug, answered product bug: 3 of 3 jobs (100%) 3 3 of 3 called product bug test issue 7 jobs Gemma at Q8, serial console: labelled test issue, answered product bug: 6 of 7 jobs (86%) 6 Gemma at Q8, serial console: labelled test issue, answered infrastructure: 1 of 7 jobs (14%) 1 6 of 7 called product bug, none a test issue product bug 3 jobs IQ2_M, serial console: labelled product bug, answered product bug: 2 of 3 jobs (67%) 2 IQ2_M, serial console: labelled product bug, answered infrastructure: 1 of 3 jobs (33%) 1 test issue 7 jobs IQ2_M, serial console: labelled test issue, answered product bug: 6 of 7 jobs (86%) 6 IQ2_M, serial console: labelled test issue, answered test issue: 1 of 7 jobs (14%) 1 1 of 7 called test issue The silver set has no serial console split.
Figure 6. Each row is a group of jobs whose class has a label; the colours show what the model answered. A model that was always right would make the product-bug rows all blue and the test-issue rows all orange. The last row is what production posted on the 111 jobs the frontier labeller called infrastructure. The buttons filter by how the failure showed up: of 13 test issues that showed up as a failing script, Gemma at Q8 called none a test issue.
Data
What the model answered for each label, by kind of failure. Gemma at Q8 and IQ2_M: the gold jobs, labels set by people, the model run again on them at 8-bit (Q8) and at the IQ2_M quantisation that runs in production; no gold job is labelled infrastructure. Silver, posted: silver jobs labelled infrastructure afterwards, against the class posted in production; the silver set has no serial-console split.
AnswersFailure kindLabelAnswered product bugAnswered test issueAnswered infrastructureJobs
Gemma at Q8all jobsproduct bug445049
Gemma at Q8all jobstest issue396146
Gemma at Q8failed in a scriptproduct bug220022
Gemma at Q8failed in a scripttest issue130013
Gemma at Q8needle mismatchproduct bug155020
Gemma at Q8needle mismatchtest issue186024
Gemma at Q8serial consoleproduct bug3003
Gemma at Q8serial consoletest issue6017
Gemma at Q8otherproduct bug4004
Gemma at Q8othertest issue2002
IQ2_Mall jobsproduct bug3513149
IQ2_Mall jobstest issue3115046
IQ2_Mfailed in a scriptproduct bug202022
IQ2_Mfailed in a scripttest issue85013
IQ2_Mneedle mismatchproduct bug911020
IQ2_Mneedle mismatchtest issue159024
IQ2_Mserial consoleproduct bug2013
IQ2_Mserial consoletest issue6107
IQ2_Motherproduct bug4004
IQ2_Mothertest issue2002
Silver, postedall jobsinfrastructure334533111
Silver, postedfailed in a scriptinfrastructure14172253
Silver, postedneedle mismatchinfrastructure720229
Silver, postedotherinfrastructure128929

A much larger model did no better that I could see. A frontier model given exactly Gemma’s input, with the written class definitions, was right on 65% of 26 gold tickets, averaged per ticket, against 60% for Gemma as production runs it.

What I tried

One run of ten prompt variants over the 116 labelled jobs I tuned prompts on took seven and a half hours, so I screened ideas on a small sample instead: 71 jobs with the three classes in equal parts, split into halves A and B. 29 of the 71 came from an internal instance, so for this set I give counts only.

The pass rule was written down before each run: at least three more jobs right than the comparison run, and no class losing two. An idea ran on A first, and on B only if it passed.

In two days I logged 179 comparisons: prompt wordings, added facts, sampling with a vote, other models. 31 passed and 148 did not. Of the 19 that passed on A and were then run on B, 18 failed there. That is what you see when the pass rule sits inside the run-to-run noise.

Net jobs gained against the control for every screened idea, by idea family Of the 19 comparisons that passed on half A and were then run on half B against the same control in a later run, 18 failed there, and 29 of the 164 comparisons on the two halves passed their rule. −10 −5 0 5 10 15 net jobs gained against the control quote the evidence · half A · 36 jobs · net 0 · dropped · product 0, test 0, infra 0 · against fuller class definitions, Gemma Q8 definitions per kind of failure · half B · 35 jobs · net +2 · dropped · product −3, test +1, infra +4 · against fuller class definitions, Gemma Q8 definitions per kind of failure · half B · 35 jobs · net −3 · dropped · product −4, test 0, infra +1 · against decision model with the environment question production wording · half A · 36 jobs · net −3 · dropped · product +1, test −2, infra −2 · against fuller class definitions, Gemma Q8 production wording · half B · 35 jobs · net −1 · dropped · product 0, test −1, infra 0 · against fuller class definitions, Gemma Q8 no example · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no example · half B · 35 jobs · net +1 · dropped · product +1, test −1, infra +1 · against fuller class definitions, Gemma Q8 step by step · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 few-shot examples · half B · 35 jobs · net −1 · dropped · product −1, test 0, infra 0 · against fuller class definitions, Gemma Q8 direct answer · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against fuller class definitions, Gemma Q8 direct answer · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 may abstain · half A · 36 jobs · net −2 · dropped · product +1, test −2, infra −1 · against fuller class definitions, Gemma Q8 may abstain · half B · 35 jobs · net −2 · dropped · product −3, test +1, infra 0 · against fuller class definitions, Gemma Q8 challenge the first answer · half A · 36 jobs · net +2 · dropped · product +1, test 0, infra +1 · against fuller class definitions, Gemma Q8 challenge the first answer · half B · 35 jobs · net +1 · dropped · product 0, test +1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net +1 · dropped · product +1, test −1, infra +1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net −6 · dropped · product 0, test −3, infra −3 · against decision model with the environment question contrast pairs in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test −1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net +4 · dropped · product +1, test +2, infra +1 · against production prompt, Gemma Q8 contrast pairs in the first prompt · half B · 35 jobs · net 0 · dropped · product 0, test +1, infra −1 · against production prompt, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net 0 · dropped · product +2, test −1, infra −1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net 0 · dropped · product +1, test −2, infra +1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net −12 · dropped · product 0, test −4, infra −8 · against decision model with the environment question no note · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no note, abstentions filled · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no note, abstentions filled · half A · 36 jobs · net −8 · dropped · product 0, test −4, infra −4 · against decision model with the environment question reasoning first, abstentions filled · half A · 36 jobs · net −4 · dropped · product −1, test +1, infra −4 · against decision model with the environment question name where it failed · half A · 36 jobs · net +2 · dropped · product +2, test 0, infra 0 · against fuller class definitions, Gemma Q8 name where it failed, abstentions filled · half A · 36 jobs · net +2 · dropped · product +2, test 0, infra 0 · against fuller class definitions, Gemma Q8 name where it failed, abstentions filled · half A · 36 jobs · net −5 · dropped · product +1, test −2, infra −4 · against decision model with the environment question proof required, abstentions filled · half A · 36 jobs · net −4 · dropped · product −2, test +1, infra −3 · against decision model with the environment question debate · half A · 36 jobs · net −3 · dropped · product −1, test −3, infra +1 · against fuller class definitions, Gemma Q8 debate, abstentions filled · half A · 36 jobs · net −3 · dropped · product −1, test −3, infra +1 · against fuller class definitions, Gemma Q8 debate, abstentions filled · half A · 36 jobs · net −10 · dropped · product −2, test −5, infra −3 · against decision model with the environment question three views · half A · 36 jobs · net −2 · dropped · product −2, test 0, infra 0 · against fuller class definitions, Gemma Q8 three views, abstentions filled · half A · 36 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 three views, abstentions filled · half A · 36 jobs · net −7 · dropped · product −2, test −1, infra −4 · against decision model with the environment question class probabilities · half A · 36 jobs · net +1 · dropped · product −3, test +4, infra 0 · against fuller class definitions, Gemma Q8 class probabilities, abstentions filled · half A · 36 jobs · net +1 · dropped · product −3, test +4, infra 0 · against fuller class definitions, Gemma Q8 class probabilities, abstentions filled · half A · 36 jobs · net −6 · dropped · product −4, test +2, infra −4 · against decision model with the environment question yes/no chain · half A · 36 jobs · net −5 · dropped · product −1, test −2, infra −2 · against fuller class definitions, Gemma Q8 yes/no chain, abstentions filled · half A · 36 jobs · net −5 · dropped · product −1, test −2, infra −2 · against fuller class definitions, Gemma Q8 yes/no chain, abstentions filled · half A · 36 jobs · net −12 · dropped · product −2, test −4, infra −6 · against decision model with the environment question reasoning first · half B · 35 jobs · net 0 · dropped · product −3, test +2, infra +1 · against fuller class definitions, Gemma Q8 reasoning first · both halves (A+B) · 71 jobs · net +3 · dropped · product −3, test +5, infra +1 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · half B · 35 jobs · net 0 · dropped · product −3, test +2, infra +1 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · both halves (A+B) · 71 jobs · net −9 · dropped · product −5, test +2, infra −6 · against decision model with the environment question proof required · half B · 35 jobs · net 0 · dropped · product −3, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required · both halves (A+B) · 71 jobs · net +3 · dropped · product −4, test +6, infra +1 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · half B · 35 jobs · net 0 · dropped · product −3, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · both halves (A+B) · 71 jobs · net −9 · dropped · product −6, test +3, infra −6 · against decision model with the environment question definitions per kind of failure · half A · 36 jobs · net +5 · passed · product −1, test +1, infra +5 · against fuller class definitions, Gemma Q8 step by step · half A · 36 jobs · net +3 · passed · product +1, test +2, infra 0 · against fuller class definitions, Gemma Q8 few-shot examples · half A · 36 jobs · net +5 · passed · product +4, test +1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net +4 · passed · product +1, test +1, infra +2 · against production prompt, Gemma Q8 reasoning first · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required · half A · 36 jobs · net +3 · passed · product −1, test +3, infra +1 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · half A · 36 jobs · net +3 · passed · product −1, test +3, infra +1 · against fuller class definitions, Gemma Q8 decision model on a focused excerpt · half A · 36 jobs · net −3 · dropped · product −1, test 0, infra −2 · against decision model with the environment question decision model on Gemma's analysis only · half A · 36 jobs · net −8 · dropped · product +1, test −5, infra −4 · against decision model with the environment question decision model on the excerpt plus Gemma's analysis · half A · 36 jobs · net 0 · dropped · product 0, test +2, infra −2 · against decision model with the environment question job facts listed · half A · 36 jobs · net −1 · dropped · product +2, test −3, infra 0 · against fuller class definitions, Gemma Q8 job facts listed · half B · 35 jobs · net −1 · dropped · product −2, test +1, infra 0 · against fuller class definitions, Gemma Q8 decision model's estimate as a hint · half A · 36 jobs · net +2 · dropped · product +1, test +1, infra 0 · against fuller class definitions, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test −1, infra 0 · against fuller class definitions, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against production prompt, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net −6 · dropped · product +1, test −3, infra −4 · against decision model with the environment question job statistics in the first prompt · half A · 36 jobs · net −2 · dropped · product +2, test −2, infra −2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net −3 · dropped · product +2, test −3, infra −2 · against decision model with the environment question test source in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test 0, infra −1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +2 · dropped · product +1, test +1, infra 0 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net −5 · dropped · product 0, test −1, infra −4 · against decision model with the environment question decision model's area as a steer · half A · 36 jobs · net −2 · dropped · product +1, test 0, infra −3 · against decision model with the environment question decision model's area as a steer · half B · 35 jobs · net +2 · dropped · product 0, test +2, infra 0 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net −2 · dropped · product −1, test 0, infra −1 · against decision model's estimate as a hint, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net −3 · dropped · product −1, test +1, infra −3 · against decision model with the environment question decision model's area as a steer · both halves (A+B) · 71 jobs · net −5 · dropped · product 0, test +1, infra −6 · against decision model with the environment question decision model's area as a steer · both halves (A+B) · 71 jobs · net +3 · dropped · product 0, test +2, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net +1 · dropped · product −1, test +1, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · both halves (A+B) · 71 jobs · net +1 · dropped · product 0, test +1, infra 0 · against decision model's estimate as a hint, Gemma Q8 test source in the first prompt · half B · 35 jobs · net 0 · dropped · product 0, test 0, infra 0 · against fuller class definitions, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net +4 · dropped · product +3, test −1, infra +2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half B · 35 jobs · net −1 · dropped · product +1, test 0, infra −2 · against production prompt, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net +3 · dropped · product +2, test +2, infra −1 · against production prompt, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net −8 · dropped · product +1, test −4, infra −5 · against decision model with the environment question test source in the first prompt · both halves (A+B) · 71 jobs · net −2 · dropped · product +1, test −4, infra +1 · against decision model's estimate as a hint, Gemma Q8 steer plus test source · half A · 36 jobs · net −3 · dropped · product −1, test −2, infra 0 · against steered analysis, production verdict steer plus test source · half A · 36 jobs · net +2 · dropped · product +2, test −1, infra +1 · against fuller class definitions, Gemma Q8 steer plus test source · half A · 36 jobs · net 0 · dropped · product +1, test −2, infra +1 · against steered analysis, definitions verdict steer plus test source · half A · 36 jobs · net −5 · dropped · product 0, test −2, infra −3 · against decision model with the environment question steer plus test source · half A · 36 jobs · net −5 · dropped · product +1, test −3, infra −3 · against decision model with the environment question steer plus test source · half A · 36 jobs · net +1 · dropped · product −1, test 0, infra +2 · against test source in the prompt, production verdict steer plus test source · half A · 36 jobs · net −2 · dropped · product −1, test 0, infra −1 · against test source in the prompt, definitions verdict decision model's estimate as a hint · half B · 35 jobs · net +4 · passed · product +1, test +2, infra +1 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net +4 · passed · product +3, test −1, infra +2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net +4 · passed · product +1, test +2, infra +1 · against production prompt, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +8 · passed · product +1, test +4, infra +3 · against production prompt, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +5 · passed · product +2, test +2, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +3 · passed · product +1, test +1, infra +1 · against decision model's estimate as a hint, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +4 · passed · product 0, test +2, infra +2 · against test source in the prompt, production verdict decision model's area as a steer · both halves (A+B) · 71 jobs · net +11 · passed · product +1, test +7, infra +3 · against production prompt, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net +3 · passed · product 0, test +3, infra 0 · against production prompt, Gemma Q8 steer plus test source · half A · 36 jobs · net +5 · passed · product 0, test +2, infra +3 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · half A · 36 jobs · net +8 · dropped (class rule: product down by 2 (the rule allows 1 on A or B)) · product −2, test +7, infra +3 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · half A · 36 jobs · net −2 · dropped · product −2, test +3, infra −3 · against decision model with the environment question gpt-oss-20b, definitions · half A · 36 jobs · net 0 · dropped · product −1, test +4, infra −3 · against decision model with the environment question gpt-oss-20b, with hint · half A · 36 jobs · net −1 · dropped · product 0, test +2, infra −3 · against decision model with the environment question Gemma-26B-A4B, production wording · half A · 36 jobs · net +2 · dropped · product +3, test 0, infra −1 · against decision model with the environment question Gemma-26B-A4B, definitions · half A · 36 jobs · net −2 · dropped · product +3, test −3, infra −2 · against decision model with the environment question Gemma-26B-A4B, with hint · half A · 36 jobs · net −1 · dropped · product +2, test −1, infra −2 · against decision model with the environment question gpt-oss-20b, definitions · half B · 35 jobs · net +3 · dropped (class rule: product down by 3 (the rule allows 1 on A or B)) · product −3, test +4, infra +2 · against fuller class definitions, Gemma Q8 gpt-oss-20b, definitions · both halves (A+B) · 71 jobs · net +10 · dropped (class rule: product down by 3 (the rule allows 2 on A+B)) · product −3, test +10, infra +3 · against fuller class definitions, Gemma Q8 gpt-oss-20b, with hint · half B · 35 jobs · net 0 · dropped · product −2, test 0, infra +2 · against decision model's estimate as a hint, Gemma Q8 gpt-oss-20b, with hint · both halves (A+B) · 71 jobs · net +4 · dropped · product −2, test +3, infra +3 · against decision model's estimate as a hint, Gemma Q8 gpt-oss-20b, production wording · half B · 35 jobs · net +4 · dropped (class rule: product down by 3 (the rule allows 1 on A or B)) · product −3, test +6, infra +1 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · both halves (A+B) · 71 jobs · net +12 · dropped (class rule: product down by 5 (the rule allows 2 on A+B)) · product −5, test +13, infra +4 · against production prompt, Gemma Q8 gpt-oss-20b, definitions · both halves (A+B) · 71 jobs · net −2 · dropped · product −5, test +7, infra −4 · against decision model with the environment question gpt-oss-20b, with hint · both halves (A+B) · 71 jobs · net −2 · dropped · product −2, test +3, infra −3 · against decision model with the environment question Gemma-26B-A4B, production wording · half B · 32 jobs · net +3 · dropped (coverage: 3 of 35 jobs timed out, so it was scored on 32 only and not counted as a pass) · product +1, test 0, infra +2 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · half B · 32 jobs · net +2 · dropped · product +1, test −1, infra +2 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · half B · 32 jobs · net −4 · dropped · product −2, test −3, infra +1 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · both halves (A+B) · 68 jobs · net +15 · dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) · product +4, test +4, infra +7 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · both halves (A+B) · 68 jobs · net +7 · dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) · product +5, test −2, infra +4 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · both halves (A+B) · 68 jobs · net 0 · dropped · product 0, test −3, infra +3 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · both halves (A+B) · 68 jobs · net −1 · dropped · product +3, test −2, infra −2 · against decision model with the environment question Gemma-26B-A4B, production wording · half B · 32 jobs · net −3 · dropped · product 0, test −2, infra −1 · against decision model with the environment question gpt-oss-20b, definitions · half A · 36 jobs · net +7 · passed · product 0, test +6, infra +1 · against fuller class definitions, Gemma Q8 gpt-oss-20b, with hint · half A · 36 jobs · net +4 · passed · product 0, test +3, infra +1 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · half A · 36 jobs · net +12 · passed · product +3, test +4, infra +5 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · half A · 36 jobs · net +5 · passed · product +4, test −1, infra +2 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · half A · 36 jobs · net +4 · passed · product +2, test 0, infra +2 · against decision model's estimate as a hint, Gemma Q8 Kev-9B, environment rule · half A · 36 jobs · net −10 · dropped · product −8, test −5, infra +3 · against decision model with the environment question Kev-4B, environment rule · half A · 36 jobs · net −8 · dropped · product −6, test −5, infra +3 · against decision model with the environment question Kev-4B, label question · half A · 36 jobs · net −7 · dropped · product −8, test −5, infra +6 · against decision model, label question only Clef-Flash, environment rule · half A · 36 jobs · net −3 · dropped · product −4, test +4, infra −3 · against decision model with the environment question Clef-Flash, label question · half A · 36 jobs · net −2 · dropped · product −5, test +3, infra 0 · against decision model, label question only d3-flash, environment rule · half A · 36 jobs · net −3 · dropped · product −5, test +4, infra −2 · against decision model with the environment question d3-flash, label question · half A · 36 jobs · net −2 · dropped · product −5, test +4, infra −1 · against decision model, label question only Kev-9B, label question · half B · 35 jobs · net −2 · dropped · product −4, test +1, infra +1 · against decision model, label question only Kev-9B, environment rule · half B · 35 jobs · net −7 · dropped · product −9, test −4, infra +6 · against decision model with the environment question Kev-9B, label question · half B · 35 jobs · net −5 · dropped · product −5, test +1, infra −1 · against decision model with the environment question Kev-9B, label question · both halves (A+B) · 71 jobs · net +1 · dropped · product −5, test +1, infra +5 · against decision model, label question only Kev-27B, environment rule · half A · 36 jobs · net +2 · dropped · product +1, test 0, infra +1 · against decision model with the environment question Kev-27B, label question · half B · 35 jobs · net +4 · dropped (class rule: product down by 2 (the rule allows 1 on A or B)) · product −2, test 0, infra +6 · against decision model, label question only Kev-27B, environment rule · half B · 35 jobs · net +1 · dropped · product −3, test 0, infra +4 · against decision model with the environment question Kev-27B, environment rule · both halves (A+B) · 71 jobs · net +3 · dropped · product −2, test 0, infra +5 · against decision model with the environment question Kev-9B, label question · half A · 36 jobs · net +3 · passed · product −1, test 0, infra +4 · against decision model, label question only Kev-27B, label question · half A · 36 jobs · net +5 · passed · product +1, test 0, infra +4 · against decision model, label question only Kev-27B, label question · both halves (A+B) · 71 jobs · net +9 · passed · product −1, test 0, infra +10 · against decision model, label question only hint plus environment override · half B · 33 jobs · net +2 · dropped · product +1, test +1, infra 0 · against decision model with the environment question hint plus environment override · half A · 33 jobs · net +1 · dropped · product +1, test 0, infra 0 · against decision model with the environment question nearest labelled precedent rule · half A · 36 jobs · net −1 · dropped · product 0, test −1, infra 0 · against decision model with the environment question nearest labelled precedent rule · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against decision model with the environment question Kev-27B and elephant-1 averaged · half A · 36 jobs · net −1 · dropped · product −1, test −1, infra +1 · against decision model with the environment question Kev-27B and elephant-1 averaged · both halves (A+B) · 71 jobs · net +2 · dropped · product −2, test −1, infra +5 · against decision model with the environment question Kev-27B and elephant-1 averaged · half B · 35 jobs · net +3 · passed · product −1, test 0, infra +4 · against decision model with the environment question verdict asked again · half A · 36 jobs · net 0 · dropped · product +1, test −1, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half A · 36 jobs · net −4 · dropped · product −1, test +1, infra −4 · against decision model with the environment question verdict in a fresh conversation · both halves (A+B) · 71 jobs · net +3 · dropped · product −1, test +4, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 majority of five samples · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 TF-IDF baseline · half A · 36 jobs · net +9 · passed (passed A, then a leakage check showed it had learnt to recognise tickets (jobs of one ticket on both sides of its split); trained only on earlier jobs it was level with the decision rule) · product +2, test +4, infra +3 · against decision model with the environment question verdict passed its rule dropped screen half A half B both halves (A+B) 18 of the 19 comparisons that passed on half A failed when run on half B against the same control in a later run. A line joins each such pass to its half-B run. 29 of the 164 comparisons drawn passed their rule. +3: rule for one half +5: rule for both halves prompt wording 59 tries added facts 45 tries bigger chat models 34 tries, 28 drawn other decision models 19 tries, 18 drawn combination rules 11 tries, 7 drawn sampling and voting 6 tries flags 3 tries 3 tries with no net count classic baseline 2 tries, 1 drawn contrast pairs on A+B: −12 few-shot examples, one run: +5 on half A, −1 on half B Gemma-26B-A4B: +15 on 68 of 71, 3 timed out, not counted TF-IDF baseline: had learnt to recognise tickets
Net jobs gained against the control for every screened idea, by idea family Of the 19 comparisons that passed on half A and were then run on half B against the same control in a later run, 18 failed there, and 29 of the 164 comparisons on the two halves passed their rule. −10 −5 0 5 10 15 net jobs gained against the control quote the evidence · half A · 36 jobs · net 0 · dropped · product 0, test 0, infra 0 · against fuller class definitions, Gemma Q8 definitions per kind of failure · half B · 35 jobs · net +2 · dropped · product −3, test +1, infra +4 · against fuller class definitions, Gemma Q8 definitions per kind of failure · half B · 35 jobs · net −3 · dropped · product −4, test 0, infra +1 · against decision model with the environment question production wording · half A · 36 jobs · net −3 · dropped · product +1, test −2, infra −2 · against fuller class definitions, Gemma Q8 production wording · half B · 35 jobs · net −1 · dropped · product 0, test −1, infra 0 · against fuller class definitions, Gemma Q8 no example · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no example · half B · 35 jobs · net +1 · dropped · product +1, test −1, infra +1 · against fuller class definitions, Gemma Q8 step by step · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 few-shot examples · half B · 35 jobs · net −1 · dropped · product −1, test 0, infra 0 · against fuller class definitions, Gemma Q8 direct answer · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against fuller class definitions, Gemma Q8 direct answer · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 may abstain · half A · 36 jobs · net −2 · dropped · product +1, test −2, infra −1 · against fuller class definitions, Gemma Q8 may abstain · half B · 35 jobs · net −2 · dropped · product −3, test +1, infra 0 · against fuller class definitions, Gemma Q8 challenge the first answer · half A · 36 jobs · net +2 · dropped · product +1, test 0, infra +1 · against fuller class definitions, Gemma Q8 challenge the first answer · half B · 35 jobs · net +1 · dropped · product 0, test +1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net +1 · dropped · product +1, test −1, infra +1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net −6 · dropped · product 0, test −3, infra −3 · against decision model with the environment question contrast pairs in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test −1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net +4 · dropped · product +1, test +2, infra +1 · against production prompt, Gemma Q8 contrast pairs in the first prompt · half B · 35 jobs · net 0 · dropped · product 0, test +1, infra −1 · against production prompt, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net 0 · dropped · product +2, test −1, infra −1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net 0 · dropped · product +1, test −2, infra +1 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · both halves (A+B) · 71 jobs · net −12 · dropped · product 0, test −4, infra −8 · against decision model with the environment question no note · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no note, abstentions filled · half A · 36 jobs · net −1 · dropped · product +1, test −2, infra 0 · against fuller class definitions, Gemma Q8 no note, abstentions filled · half A · 36 jobs · net −8 · dropped · product 0, test −4, infra −4 · against decision model with the environment question reasoning first, abstentions filled · half A · 36 jobs · net −4 · dropped · product −1, test +1, infra −4 · against decision model with the environment question name where it failed · half A · 36 jobs · net +2 · dropped · product +2, test 0, infra 0 · against fuller class definitions, Gemma Q8 name where it failed, abstentions filled · half A · 36 jobs · net +2 · dropped · product +2, test 0, infra 0 · against fuller class definitions, Gemma Q8 name where it failed, abstentions filled · half A · 36 jobs · net −5 · dropped · product +1, test −2, infra −4 · against decision model with the environment question proof required, abstentions filled · half A · 36 jobs · net −4 · dropped · product −2, test +1, infra −3 · against decision model with the environment question debate · half A · 36 jobs · net −3 · dropped · product −1, test −3, infra +1 · against fuller class definitions, Gemma Q8 debate, abstentions filled · half A · 36 jobs · net −3 · dropped · product −1, test −3, infra +1 · against fuller class definitions, Gemma Q8 debate, abstentions filled · half A · 36 jobs · net −10 · dropped · product −2, test −5, infra −3 · against decision model with the environment question three views · half A · 36 jobs · net −2 · dropped · product −2, test 0, infra 0 · against fuller class definitions, Gemma Q8 three views, abstentions filled · half A · 36 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 three views, abstentions filled · half A · 36 jobs · net −7 · dropped · product −2, test −1, infra −4 · against decision model with the environment question class probabilities · half A · 36 jobs · net +1 · dropped · product −3, test +4, infra 0 · against fuller class definitions, Gemma Q8 class probabilities, abstentions filled · half A · 36 jobs · net +1 · dropped · product −3, test +4, infra 0 · against fuller class definitions, Gemma Q8 class probabilities, abstentions filled · half A · 36 jobs · net −6 · dropped · product −4, test +2, infra −4 · against decision model with the environment question yes/no chain · half A · 36 jobs · net −5 · dropped · product −1, test −2, infra −2 · against fuller class definitions, Gemma Q8 yes/no chain, abstentions filled · half A · 36 jobs · net −5 · dropped · product −1, test −2, infra −2 · against fuller class definitions, Gemma Q8 yes/no chain, abstentions filled · half A · 36 jobs · net −12 · dropped · product −2, test −4, infra −6 · against decision model with the environment question reasoning first · half B · 35 jobs · net 0 · dropped · product −3, test +2, infra +1 · against fuller class definitions, Gemma Q8 reasoning first · both halves (A+B) · 71 jobs · net +3 · dropped · product −3, test +5, infra +1 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · half B · 35 jobs · net 0 · dropped · product −3, test +2, infra +1 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · both halves (A+B) · 71 jobs · net −9 · dropped · product −5, test +2, infra −6 · against decision model with the environment question proof required · half B · 35 jobs · net 0 · dropped · product −3, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required · both halves (A+B) · 71 jobs · net +3 · dropped · product −4, test +6, infra +1 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · half B · 35 jobs · net 0 · dropped · product −3, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · both halves (A+B) · 71 jobs · net −9 · dropped · product −6, test +3, infra −6 · against decision model with the environment question definitions per kind of failure · half A · 36 jobs · net +5 · passed · product −1, test +1, infra +5 · against fuller class definitions, Gemma Q8 step by step · half A · 36 jobs · net +3 · passed · product +1, test +2, infra 0 · against fuller class definitions, Gemma Q8 few-shot examples · half A · 36 jobs · net +5 · passed · product +4, test +1, infra 0 · against fuller class definitions, Gemma Q8 contrast pairs in the first prompt · half A · 36 jobs · net +4 · passed · product +1, test +1, infra +2 · against production prompt, Gemma Q8 reasoning first · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 reasoning first, abstentions filled · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 proof required · half A · 36 jobs · net +3 · passed · product −1, test +3, infra +1 · against fuller class definitions, Gemma Q8 proof required, abstentions filled · half A · 36 jobs · net +3 · passed · product −1, test +3, infra +1 · against fuller class definitions, Gemma Q8 decision model on a focused excerpt · half A · 36 jobs · net −3 · dropped · product −1, test 0, infra −2 · against decision model with the environment question decision model on Gemma's analysis only · half A · 36 jobs · net −8 · dropped · product +1, test −5, infra −4 · against decision model with the environment question decision model on the excerpt plus Gemma's analysis · half A · 36 jobs · net 0 · dropped · product 0, test +2, infra −2 · against decision model with the environment question job facts listed · half A · 36 jobs · net −1 · dropped · product +2, test −3, infra 0 · against fuller class definitions, Gemma Q8 job facts listed · half B · 35 jobs · net −1 · dropped · product −2, test +1, infra 0 · against fuller class definitions, Gemma Q8 decision model's estimate as a hint · half A · 36 jobs · net +2 · dropped · product +1, test +1, infra 0 · against fuller class definitions, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test −1, infra 0 · against fuller class definitions, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against production prompt, Gemma Q8 job statistics in the first prompt · half A · 36 jobs · net −6 · dropped · product +1, test −3, infra −4 · against decision model with the environment question job statistics in the first prompt · half A · 36 jobs · net −2 · dropped · product +2, test −2, infra −2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net −3 · dropped · product +2, test −3, infra −2 · against decision model with the environment question test source in the first prompt · half A · 36 jobs · net +1 · dropped · product +2, test 0, infra −1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +2 · dropped · product +1, test +1, infra 0 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net −5 · dropped · product 0, test −1, infra −4 · against decision model with the environment question decision model's area as a steer · half A · 36 jobs · net −2 · dropped · product +1, test 0, infra −3 · against decision model with the environment question decision model's area as a steer · half B · 35 jobs · net +2 · dropped · product 0, test +2, infra 0 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net −2 · dropped · product −1, test 0, infra −1 · against decision model's estimate as a hint, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net −3 · dropped · product −1, test +1, infra −3 · against decision model with the environment question decision model's area as a steer · both halves (A+B) · 71 jobs · net −5 · dropped · product 0, test +1, infra −6 · against decision model with the environment question decision model's area as a steer · both halves (A+B) · 71 jobs · net +3 · dropped · product 0, test +2, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net +1 · dropped · product −1, test +1, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · both halves (A+B) · 71 jobs · net +1 · dropped · product 0, test +1, infra 0 · against decision model's estimate as a hint, Gemma Q8 test source in the first prompt · half B · 35 jobs · net 0 · dropped · product 0, test 0, infra 0 · against fuller class definitions, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net +4 · dropped · product +3, test −1, infra +2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half B · 35 jobs · net −1 · dropped · product +1, test 0, infra −2 · against production prompt, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net +3 · dropped · product +2, test +2, infra −1 · against production prompt, Gemma Q8 test source in the first prompt · both halves (A+B) · 71 jobs · net −8 · dropped · product +1, test −4, infra −5 · against decision model with the environment question test source in the first prompt · both halves (A+B) · 71 jobs · net −2 · dropped · product +1, test −4, infra +1 · against decision model's estimate as a hint, Gemma Q8 steer plus test source · half A · 36 jobs · net −3 · dropped · product −1, test −2, infra 0 · against steered analysis, production verdict steer plus test source · half A · 36 jobs · net +2 · dropped · product +2, test −1, infra +1 · against fuller class definitions, Gemma Q8 steer plus test source · half A · 36 jobs · net 0 · dropped · product +1, test −2, infra +1 · against steered analysis, definitions verdict steer plus test source · half A · 36 jobs · net −5 · dropped · product 0, test −2, infra −3 · against decision model with the environment question steer plus test source · half A · 36 jobs · net −5 · dropped · product +1, test −3, infra −3 · against decision model with the environment question steer plus test source · half A · 36 jobs · net +1 · dropped · product −1, test 0, infra +2 · against test source in the prompt, production verdict steer plus test source · half A · 36 jobs · net −2 · dropped · product −1, test 0, infra −1 · against test source in the prompt, definitions verdict decision model's estimate as a hint · half B · 35 jobs · net +4 · passed · product +1, test +2, infra +1 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net +4 · passed · product +3, test −1, infra +2 · against fuller class definitions, Gemma Q8 test source in the first prompt · half A · 36 jobs · net +4 · passed · product +1, test +2, infra +1 · against production prompt, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +8 · passed · product +1, test +4, infra +3 · against production prompt, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +5 · passed · product +2, test +2, infra +1 · against fuller class definitions, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +3 · passed · product +1, test +1, infra +1 · against decision model's estimate as a hint, Gemma Q8 decision model's area as a steer · half A · 36 jobs · net +4 · passed · product 0, test +2, infra +2 · against test source in the prompt, production verdict decision model's area as a steer · both halves (A+B) · 71 jobs · net +11 · passed · product +1, test +7, infra +3 · against production prompt, Gemma Q8 decision model's area as a steer · half B · 35 jobs · net +3 · passed · product 0, test +3, infra 0 · against production prompt, Gemma Q8 steer plus test source · half A · 36 jobs · net +5 · passed · product 0, test +2, infra +3 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · half A · 36 jobs · net +8 · dropped (class rule: product down by 2 (the rule allows 1 on A or B)) · product −2, test +7, infra +3 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · half A · 36 jobs · net −2 · dropped · product −2, test +3, infra −3 · against decision model with the environment question gpt-oss-20b, definitions · half A · 36 jobs · net 0 · dropped · product −1, test +4, infra −3 · against decision model with the environment question gpt-oss-20b, with hint · half A · 36 jobs · net −1 · dropped · product 0, test +2, infra −3 · against decision model with the environment question Gemma-26B-A4B, production wording · half A · 36 jobs · net +2 · dropped · product +3, test 0, infra −1 · against decision model with the environment question Gemma-26B-A4B, definitions · half A · 36 jobs · net −2 · dropped · product +3, test −3, infra −2 · against decision model with the environment question Gemma-26B-A4B, with hint · half A · 36 jobs · net −1 · dropped · product +2, test −1, infra −2 · against decision model with the environment question gpt-oss-20b, definitions · half B · 35 jobs · net +3 · dropped (class rule: product down by 3 (the rule allows 1 on A or B)) · product −3, test +4, infra +2 · against fuller class definitions, Gemma Q8 gpt-oss-20b, definitions · both halves (A+B) · 71 jobs · net +10 · dropped (class rule: product down by 3 (the rule allows 2 on A+B)) · product −3, test +10, infra +3 · against fuller class definitions, Gemma Q8 gpt-oss-20b, with hint · half B · 35 jobs · net 0 · dropped · product −2, test 0, infra +2 · against decision model's estimate as a hint, Gemma Q8 gpt-oss-20b, with hint · both halves (A+B) · 71 jobs · net +4 · dropped · product −2, test +3, infra +3 · against decision model's estimate as a hint, Gemma Q8 gpt-oss-20b, production wording · half B · 35 jobs · net +4 · dropped (class rule: product down by 3 (the rule allows 1 on A or B)) · product −3, test +6, infra +1 · against production prompt, Gemma Q8 gpt-oss-20b, production wording · both halves (A+B) · 71 jobs · net +12 · dropped (class rule: product down by 5 (the rule allows 2 on A+B)) · product −5, test +13, infra +4 · against production prompt, Gemma Q8 gpt-oss-20b, definitions · both halves (A+B) · 71 jobs · net −2 · dropped · product −5, test +7, infra −4 · against decision model with the environment question gpt-oss-20b, with hint · both halves (A+B) · 71 jobs · net −2 · dropped · product −2, test +3, infra −3 · against decision model with the environment question Gemma-26B-A4B, production wording · half B · 32 jobs · net +3 · dropped (coverage: 3 of 35 jobs timed out, so it was scored on 32 only and not counted as a pass) · product +1, test 0, infra +2 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · half B · 32 jobs · net +2 · dropped · product +1, test −1, infra +2 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · half B · 32 jobs · net −4 · dropped · product −2, test −3, infra +1 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · both halves (A+B) · 68 jobs · net +15 · dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) · product +4, test +4, infra +7 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · both halves (A+B) · 68 jobs · net +7 · dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) · product +5, test −2, infra +4 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · both halves (A+B) · 68 jobs · net 0 · dropped · product 0, test −3, infra +3 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · both halves (A+B) · 68 jobs · net −1 · dropped · product +3, test −2, infra −2 · against decision model with the environment question Gemma-26B-A4B, production wording · half B · 32 jobs · net −3 · dropped · product 0, test −2, infra −1 · against decision model with the environment question gpt-oss-20b, definitions · half A · 36 jobs · net +7 · passed · product 0, test +6, infra +1 · against fuller class definitions, Gemma Q8 gpt-oss-20b, with hint · half A · 36 jobs · net +4 · passed · product 0, test +3, infra +1 · against decision model's estimate as a hint, Gemma Q8 Gemma-26B-A4B, production wording · half A · 36 jobs · net +12 · passed · product +3, test +4, infra +5 · against production prompt, Gemma Q8 Gemma-26B-A4B, definitions · half A · 36 jobs · net +5 · passed · product +4, test −1, infra +2 · against fuller class definitions, Gemma Q8 Gemma-26B-A4B, with hint · half A · 36 jobs · net +4 · passed · product +2, test 0, infra +2 · against decision model's estimate as a hint, Gemma Q8 Kev-9B, environment rule · half A · 36 jobs · net −10 · dropped · product −8, test −5, infra +3 · against decision model with the environment question Kev-4B, environment rule · half A · 36 jobs · net −8 · dropped · product −6, test −5, infra +3 · against decision model with the environment question Kev-4B, label question · half A · 36 jobs · net −7 · dropped · product −8, test −5, infra +6 · against decision model, label question only Clef-Flash, environment rule · half A · 36 jobs · net −3 · dropped · product −4, test +4, infra −3 · against decision model with the environment question Clef-Flash, label question · half A · 36 jobs · net −2 · dropped · product −5, test +3, infra 0 · against decision model, label question only d3-flash, environment rule · half A · 36 jobs · net −3 · dropped · product −5, test +4, infra −2 · against decision model with the environment question d3-flash, label question · half A · 36 jobs · net −2 · dropped · product −5, test +4, infra −1 · against decision model, label question only Kev-9B, label question · half B · 35 jobs · net −2 · dropped · product −4, test +1, infra +1 · against decision model, label question only Kev-9B, environment rule · half B · 35 jobs · net −7 · dropped · product −9, test −4, infra +6 · against decision model with the environment question Kev-9B, label question · half B · 35 jobs · net −5 · dropped · product −5, test +1, infra −1 · against decision model with the environment question Kev-9B, label question · both halves (A+B) · 71 jobs · net +1 · dropped · product −5, test +1, infra +5 · against decision model, label question only Kev-27B, environment rule · half A · 36 jobs · net +2 · dropped · product +1, test 0, infra +1 · against decision model with the environment question Kev-27B, label question · half B · 35 jobs · net +4 · dropped (class rule: product down by 2 (the rule allows 1 on A or B)) · product −2, test 0, infra +6 · against decision model, label question only Kev-27B, environment rule · half B · 35 jobs · net +1 · dropped · product −3, test 0, infra +4 · against decision model with the environment question Kev-27B, environment rule · both halves (A+B) · 71 jobs · net +3 · dropped · product −2, test 0, infra +5 · against decision model with the environment question Kev-9B, label question · half A · 36 jobs · net +3 · passed · product −1, test 0, infra +4 · against decision model, label question only Kev-27B, label question · half A · 36 jobs · net +5 · passed · product +1, test 0, infra +4 · against decision model, label question only Kev-27B, label question · both halves (A+B) · 71 jobs · net +9 · passed · product −1, test 0, infra +10 · against decision model, label question only hint plus environment override · half B · 33 jobs · net +2 · dropped · product +1, test +1, infra 0 · against decision model with the environment question hint plus environment override · half A · 33 jobs · net +1 · dropped · product +1, test 0, infra 0 · against decision model with the environment question nearest labelled precedent rule · half A · 36 jobs · net −1 · dropped · product 0, test −1, infra 0 · against decision model with the environment question nearest labelled precedent rule · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against decision model with the environment question Kev-27B and elephant-1 averaged · half A · 36 jobs · net −1 · dropped · product −1, test −1, infra +1 · against decision model with the environment question Kev-27B and elephant-1 averaged · both halves (A+B) · 71 jobs · net +2 · dropped · product −2, test −1, infra +5 · against decision model with the environment question Kev-27B and elephant-1 averaged · half B · 35 jobs · net +3 · passed · product −1, test 0, infra +4 · against decision model with the environment question verdict asked again · half A · 36 jobs · net 0 · dropped · product +1, test −1, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half A · 36 jobs · net −4 · dropped · product −1, test +1, infra −4 · against decision model with the environment question verdict in a fresh conversation · both halves (A+B) · 71 jobs · net +3 · dropped · product −1, test +4, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half B · 35 jobs · net 0 · dropped · product −1, test +1, infra 0 · against fuller class definitions, Gemma Q8 majority of five samples · half A · 36 jobs · net +1 · dropped · product +1, test 0, infra 0 · against fuller class definitions, Gemma Q8 verdict in a fresh conversation · half A · 36 jobs · net +3 · passed · product 0, test +3, infra 0 · against fuller class definitions, Gemma Q8 TF-IDF baseline · half A · 36 jobs · net +9 · passed (passed A, then a leakage check showed it had learnt to recognise tickets (jobs of one ticket on both sides of its split); trained only on earlier jobs it was level with the decision rule) · product +2, test +4, infra +3 · against decision model with the environment question verdict passed its rule dropped screen half A half B both halves (A+B) 18 of the 19 comparisons that passed on half A failed when run on half B against the same control in a later run. A line joins each such pass to its half-B run. 29 of the 164 comparisons drawn passed their rule. +3: one-half rule +5: both-halves rule prompt wording 59 tries added facts 45 tries bigger chat models 34 tries, 28 drawn other decision models 19 tries, 18 drawn combination rules 11 tries, 7 drawn sampling and voting 6 tries flags 3 tries no net count classic baseline 2 tries, 1 drawn contrast pairs: −12 few-shot, one run: A +5, B −1 Gemma 26B: +15, 3 timed out TF-IDF: leakage
Figure 7. Each dot is one comparison of an idea with the run it was measured against, placed by how many more jobs the idea got right (right of 0) or fewer (left of 0). Only dots on or beyond a vertical line passed: +3 on one half, +5 on both together. A line joins a pass on half A to the later run of the same idea on half B. Drawn are the 164 of the 179 comparisons that have such a count.
Data
All 179 logged comparisons, by idea family. Rule for one half (A, 36 jobs; B, 35 jobs): net gain of at least 3 jobs and no class down by more than 1. Rule for both halves: net gain of at least 5 of 71 and no class down by more than 2. The 6 comparisons on the 12-hard-jobs screen are folded into “other” and not drawn, since their net counts right answers over three wordings (12 hard jobs: all within 70 minutes and right answers over three wordings at least the control's plus 3); 8 rows have no net count and 1 compared no jobs, and these are not drawn either. Of the drawn rows whose net met their rule's number, 9 were dropped: 6 by the class rule, 3 for coverage; 1 pass was later overturned by a leakage check. The reason is given with the verdict. The 3 dropped for coverage were logged as dropped, with the jobs that timed out as the reason; the 1 row logged as “dropped (coverage)” had no screen run and no net count, so it is not drawn. In one repeat of the last turn alone, at temperature 0, on the 36 jobs of half A, 3 answers changed and 2 of them switched between right and wrong.
IdeaFamilyScreenJobsNetChange per classVerdictControlLogged
definitions per kind of failureprompt wordinghalf A36+5product −1, test +1, infra +5passedfuller class definitions, Gemma Q82026-11-10 15:02
quote the evidenceprompt wordinghalf A360product 0, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-10 15:02
definitions per kind of failureprompt wordinghalf B35+2product −3, test +1, infra +4droppedfuller class definitions, Gemma Q82026-11-10 15:41
definitions per kind of failureprompt wordinghalf B35−3product −4, test 0, infra +1droppeddecision model with the environment question2026-11-10 15:41
production wordingprompt wordinghalf A36−3product +1, test −2, infra −2droppedfuller class definitions, Gemma Q82026-11-10 19:01
production wordingprompt wordinghalf B35−1product 0, test −1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
no exampleprompt wordinghalf A36−1product +1, test −2, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
no exampleprompt wordinghalf B35+1product +1, test −1, infra +1droppedfuller class definitions, Gemma Q82026-11-10 19:01
step by stepprompt wordinghalf A36+3product +1, test +2, infra 0passedfuller class definitions, Gemma Q82026-11-10 19:01
step by stepprompt wordinghalf B350product −1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
few-shot examplesprompt wordinghalf A36+5product +4, test +1, infra 0passedfuller class definitions, Gemma Q82026-11-10 19:01
few-shot examplesprompt wordinghalf B35−1product −1, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
direct answerprompt wordinghalf A36+1product +1, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
direct answerprompt wordinghalf B350product −1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
may abstainprompt wordinghalf A36−2product +1, test −2, infra −1droppedfuller class definitions, Gemma Q82026-11-10 19:01
may abstainprompt wordinghalf B35−2product −3, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
challenge the first answerprompt wordinghalf A36+2product +1, test 0, infra +1droppedfuller class definitions, Gemma Q82026-11-10 19:01
challenge the first answerprompt wordinghalf B35+1product 0, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
contrast pairs in the first promptprompt wordinghalf A36+1product +1, test −1, infra +1droppedfuller class definitions, Gemma Q82026-11-10 21:05
contrast pairs in the first promptprompt wordinghalf A36+4product +1, test +1, infra +2passedproduction prompt, Gemma Q82026-11-10 21:05
contrast pairs in the first promptprompt wordinghalf A36−6product 0, test −3, infra −3droppeddecision model with the environment question2026-11-10 21:05
contrast pairs in the first promptprompt wordinghalf A36+1product +2, test −1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 21:05
contrast pairs in the first promptprompt wordingboth halves (A+B)71+4product +1, test +2, infra +1droppedproduction prompt, Gemma Q82026-11-11 01:27
contrast pairs in the first promptprompt wordinghalf B350product 0, test +1, infra −1droppedproduction prompt, Gemma Q82026-11-11 01:27
contrast pairs in the first promptprompt wordingboth halves (A+B)710product +2, test −1, infra −1droppedfuller class definitions, Gemma Q82026-11-11 01:27
contrast pairs in the first promptprompt wordingboth halves (A+B)710product +1, test −2, infra +1droppedfuller class definitions, Gemma Q82026-11-11 01:27
contrast pairs in the first promptprompt wordingboth halves (A+B)71−12product 0, test −4, infra −8droppeddecision model with the environment question2026-11-11 01:27
no noteprompt wordinghalf A36−1product +1, test −2, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
no note, abstentions filledprompt wordinghalf A36−1product +1, test −2, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
no note, abstentions filledprompt wordinghalf A36−8product 0, test −4, infra −4droppeddecision model with the environment question2026-11-11 05:04
reasoning firstprompt wordinghalf A36+3product 0, test +3, infra 0passedfuller class definitions, Gemma Q82026-11-11 05:04
reasoning first, abstentions filledprompt wordinghalf A36+3product 0, test +3, infra 0passedfuller class definitions, Gemma Q82026-11-11 05:04
reasoning first, abstentions filledprompt wordinghalf A36−4product −1, test +1, infra −4droppeddecision model with the environment question2026-11-11 05:04
name where it failedprompt wordinghalf A36+2product +2, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
name where it failed, abstentions filledprompt wordinghalf A36+2product +2, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
name where it failed, abstentions filledprompt wordinghalf A36−5product +1, test −2, infra −4droppeddecision model with the environment question2026-11-11 05:04
proof requiredprompt wordinghalf A36+3product −1, test +3, infra +1passedfuller class definitions, Gemma Q82026-11-11 05:04
proof required, abstentions filledprompt wordinghalf A36+3product −1, test +3, infra +1passedfuller class definitions, Gemma Q82026-11-11 05:04
proof required, abstentions filledprompt wordinghalf A36−4product −2, test +1, infra −3droppeddecision model with the environment question2026-11-11 05:04
debateprompt wordinghalf A36−3product −1, test −3, infra +1droppedfuller class definitions, Gemma Q82026-11-11 05:04
debate, abstentions filledprompt wordinghalf A36−3product −1, test −3, infra +1droppedfuller class definitions, Gemma Q82026-11-11 05:04
debate, abstentions filledprompt wordinghalf A36−10product −2, test −5, infra −3droppeddecision model with the environment question2026-11-11 05:04
three viewsprompt wordinghalf A36−2product −2, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
three views, abstentions filledprompt wordinghalf A360product −1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
three views, abstentions filledprompt wordinghalf A36−7product −2, test −1, infra −4droppeddecision model with the environment question2026-11-11 05:04
class probabilitiesprompt wordinghalf A36+1product −3, test +4, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
class probabilities, abstentions filledprompt wordinghalf A36+1product −3, test +4, infra 0droppedfuller class definitions, Gemma Q82026-11-11 05:04
class probabilities, abstentions filledprompt wordinghalf A36−6product −4, test +2, infra −4droppeddecision model with the environment question2026-11-11 05:04
yes/no chainprompt wordinghalf A36−5product −1, test −2, infra −2droppedfuller class definitions, Gemma Q82026-11-11 05:04
yes/no chain, abstentions filledprompt wordinghalf A36−5product −1, test −2, infra −2droppedfuller class definitions, Gemma Q82026-11-11 05:04
yes/no chain, abstentions filledprompt wordinghalf A36−12product −2, test −4, infra −6droppeddecision model with the environment question2026-11-11 05:04
reasoning firstprompt wordinghalf B350product −3, test +2, infra +1droppedfuller class definitions, Gemma Q82026-11-11 06:21
reasoning firstprompt wordingboth halves (A+B)71+3product −3, test +5, infra +1droppedfuller class definitions, Gemma Q82026-11-11 06:21
reasoning first, abstentions filledprompt wordinghalf B350product −3, test +2, infra +1droppedfuller class definitions, Gemma Q82026-11-11 06:21
reasoning first, abstentions filledprompt wordingboth halves (A+B)71−9product −5, test +2, infra −6droppeddecision model with the environment question2026-11-11 06:21
proof requiredprompt wordinghalf B350product −3, test +3, infra 0droppedfuller class definitions, Gemma Q82026-11-11 06:21
proof requiredprompt wordingboth halves (A+B)71+3product −4, test +6, infra +1droppedfuller class definitions, Gemma Q82026-11-11 06:21
proof required, abstentions filledprompt wordinghalf B350product −3, test +3, infra 0droppedfuller class definitions, Gemma Q82026-11-11 06:21
proof required, abstentions filledprompt wordingboth halves (A+B)71−9product −6, test +3, infra −6droppeddecision model with the environment question2026-11-11 06:21
decision model on a focused excerptadded factshalf A36−3product −1, test 0, infra −2droppeddecision model with the environment question2026-11-10 12:19
decision model on Gemma's analysis onlyadded factshalf A36−8product +1, test −5, infra −4droppeddecision model with the environment question2026-11-10 12:42
decision model on the excerpt plus Gemma's analysisadded factshalf A360product 0, test +2, infra −2droppeddecision model with the environment question2026-11-10 13:14
job facts listedadded factshalf A36−1product +2, test −3, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
job facts listedadded factshalf B35−1product −2, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
decision model's estimate as a hintadded factshalf A36+2product +1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 19:01
decision model's estimate as a hintadded factshalf B35+4product +1, test +2, infra +1passedfuller class definitions, Gemma Q82026-11-10 19:01
job statistics in the first promptadded factshalf A36+1product +2, test −1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 21:43
job statistics in the first promptadded factshalf A36+1product +1, test 0, infra 0droppedproduction prompt, Gemma Q82026-11-10 21:43
job statistics in the first promptadded factshalf A36−6product +1, test −3, infra −4droppeddecision model with the environment question2026-11-10 21:43
job statistics in the first promptadded factshalf A36−2product +2, test −2, infra −2droppedfuller class definitions, Gemma Q82026-11-10 21:43
test source in the first promptadded factshalf A36+4product +3, test −1, infra +2passedfuller class definitions, Gemma Q82026-11-10 22:39
test source in the first promptadded factshalf A36+4product +1, test +2, infra +1passedproduction prompt, Gemma Q82026-11-10 22:39
test source in the first promptadded factshalf A36−3product +2, test −3, infra −2droppeddecision model with the environment question2026-11-10 22:39
test source in the first promptadded factshalf A36+1product +2, test 0, infra −1droppedfuller class definitions, Gemma Q82026-11-10 22:39
decision model's area as a steeradded factshalf A36+2product +1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 23:36
decision model's area as a steeradded factshalf A36+8product +1, test +4, infra +3passedproduction prompt, Gemma Q82026-11-10 23:36
decision model's area as a steeradded factshalf A36−5product 0, test −1, infra −4droppeddecision model with the environment question2026-11-10 23:36
decision model's area as a steeradded factshalf A36+5product +2, test +2, infra +1passedfuller class definitions, Gemma Q82026-11-10 23:36
decision model's area as a steeradded factshalf A36−2product +1, test 0, infra −3droppeddecision model with the environment question2026-11-10 23:36
decision model's area as a steeradded factshalf A36+3product +1, test +1, infra +1passeddecision model's estimate as a hint, Gemma Q82026-11-10 23:36
decision model's area as a steeradded factshalf A36+4product 0, test +2, infra +2passedtest source in the prompt, production verdict2026-11-10 23:36
decision model's area as a steeradded factsboth halves (A+B)71+11product +1, test +7, infra +3passedproduction prompt, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factshalf B35+3product 0, test +3, infra 0passedproduction prompt, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factshalf B35+2product 0, test +2, infra 0droppedfuller class definitions, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factshalf B35−2product −1, test 0, infra −1droppeddecision model's estimate as a hint, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factshalf B35−3product −1, test +1, infra −3droppeddecision model with the environment question2026-11-11 00:33
decision model's area as a steeradded factsboth halves (A+B)71−5product 0, test +1, infra −6droppeddecision model with the environment question2026-11-11 00:33
decision model's area as a steeradded factsboth halves (A+B)71+3product 0, test +2, infra +1droppedfuller class definitions, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factshalf B35+1product −1, test +1, infra +1droppedfuller class definitions, Gemma Q82026-11-11 00:33
decision model's area as a steeradded factsboth halves (A+B)71+1product 0, test +1, infra 0droppeddecision model's estimate as a hint, Gemma Q82026-11-11 00:33
test source in the first promptadded factshalf B350product 0, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-11 08:51
test source in the first promptadded factsboth halves (A+B)71+4product +3, test −1, infra +2droppedfuller class definitions, Gemma Q82026-11-11 08:51
test source in the first promptadded factshalf B35−1product +1, test 0, infra −2droppedproduction prompt, Gemma Q82026-11-11 08:51
test source in the first promptadded factsboth halves (A+B)71+3product +2, test +2, infra −1droppedproduction prompt, Gemma Q82026-11-11 08:51
test source in the first promptadded factsboth halves (A+B)71−8product +1, test −4, infra −5droppeddecision model with the environment question2026-11-11 08:51
test source in the first promptadded factsboth halves (A+B)71−2product +1, test −4, infra +1droppeddecision model's estimate as a hint, Gemma Q82026-11-11 08:51
steer plus test sourceadded factshalf A36+5product 0, test +2, infra +3passedproduction prompt, Gemma Q82026-11-11 09:48
steer plus test sourceadded factshalf A36−3product −1, test −2, infra 0droppedsteered analysis, production verdict2026-11-11 09:48
steer plus test sourceadded factshalf A36+2product +2, test −1, infra +1droppedfuller class definitions, Gemma Q82026-11-11 09:48
steer plus test sourceadded factshalf A360product +1, test −2, infra +1droppedsteered analysis, definitions verdict2026-11-11 09:48
steer plus test sourceadded factshalf A36−5product 0, test −2, infra −3droppeddecision model with the environment question2026-11-11 09:48
steer plus test sourceadded factshalf A36−5product +1, test −3, infra −3droppeddecision model with the environment question2026-11-11 09:48
steer plus test sourceadded factshalf A36+1product −1, test 0, infra +2droppedtest source in the prompt, production verdict2026-11-11 09:49
steer plus test sourceadded factshalf A36−2product −1, test 0, infra −1droppedtest source in the prompt, definitions verdict2026-11-11 09:49
Ornith-9B, three wordingsbigger chat models12 hard jobs (other), not drawn–+3–dropped (cut at the 70-minute limit with 4 of 12 jobs answered (time rule))Gemma Q8, three verdict wordings pooled2026-11-11 10:58
Gemma-26B-A4B, three wordingsbigger chat models12 hard jobs (other), not drawn–+8–passedGemma Q8, three verdict wordings pooled2026-11-11 11:59
Qwen3.5-35B, three wordingsbigger chat models12 hard jobs (other), not drawn–0–droppedGemma Q8, three verdict wordings pooled2026-11-11 14:21
gpt-oss-20b, three wordingsbigger chat models12 hard jobs (other), not drawn–+4–passedGemma Q8, three verdict wordings pooled2026-11-11 14:33
gpt-oss-20b, production wordingbigger chat modelshalf A36+8product −2, test +7, infra +3dropped (class rule: product down by 2 (the rule allows 1 on A or B))production prompt, Gemma Q82026-11-11 14:57
gpt-oss-20b, definitionsbigger chat modelshalf A36+7product 0, test +6, infra +1passedfuller class definitions, Gemma Q82026-11-11 14:57
gpt-oss-20b, with hintbigger chat modelshalf A36+4product 0, test +3, infra +1passeddecision model's estimate as a hint, Gemma Q82026-11-11 14:57
gpt-oss-20b, production wordingbigger chat modelshalf A36−2product −2, test +3, infra −3droppeddecision model with the environment question2026-11-11 14:57
gpt-oss-20b, definitionsbigger chat modelshalf A360product −1, test +4, infra −3droppeddecision model with the environment question2026-11-11 14:57
gpt-oss-20b, with hintbigger chat modelshalf A36−1product 0, test +2, infra −3droppeddecision model with the environment question2026-11-11 14:57
Gemma-26B-A4B, production wordingbigger chat modelshalf A36+12product +3, test +4, infra +5passedproduction prompt, Gemma Q82026-11-11 15:36
Gemma-26B-A4B, definitionsbigger chat modelshalf A36+5product +4, test −1, infra +2passedfuller class definitions, Gemma Q82026-11-11 15:36
Gemma-26B-A4B, with hintbigger chat modelshalf A36+4product +2, test 0, infra +2passeddecision model's estimate as a hint, Gemma Q82026-11-11 15:36
Gemma-26B-A4B, production wordingbigger chat modelshalf A36+2product +3, test 0, infra −1droppeddecision model with the environment question2026-11-11 15:36
Gemma-26B-A4B, definitionsbigger chat modelshalf A36−2product +3, test −3, infra −2droppeddecision model with the environment question2026-11-11 15:36
Gemma-26B-A4B, with hintbigger chat modelshalf A36−1product +2, test −1, infra −2droppeddecision model with the environment question2026-11-11 15:36
gpt-oss-20b, definitionsbigger chat modelshalf B35+3product −3, test +4, infra +2dropped (class rule: product down by 3 (the rule allows 1 on A or B))fuller class definitions, Gemma Q82026-11-11 16:10
gpt-oss-20b, definitionsbigger chat modelsboth halves (A+B)71+10product −3, test +10, infra +3dropped (class rule: product down by 3 (the rule allows 2 on A+B))fuller class definitions, Gemma Q82026-11-11 16:10
gpt-oss-20b, with hintbigger chat modelshalf B350product −2, test 0, infra +2droppeddecision model's estimate as a hint, Gemma Q82026-11-11 16:10
gpt-oss-20b, with hintbigger chat modelsboth halves (A+B)71+4product −2, test +3, infra +3droppeddecision model's estimate as a hint, Gemma Q82026-11-11 16:10
gpt-oss-20b, production wordingbigger chat modelshalf B35+4product −3, test +6, infra +1dropped (class rule: product down by 3 (the rule allows 1 on A or B))production prompt, Gemma Q82026-11-11 16:10
gpt-oss-20b, production wordingbigger chat modelsboth halves (A+B)71+12product −5, test +13, infra +4dropped (class rule: product down by 5 (the rule allows 2 on A+B))production prompt, Gemma Q82026-11-11 16:10
gpt-oss-20b, definitionsbigger chat modelsboth halves (A+B)71−2product −5, test +7, infra −4droppeddecision model with the environment question2026-11-11 16:10
gpt-oss-20b, with hintbigger chat modelsboth halves (A+B)71−2product −2, test +3, infra −3droppeddecision model with the environment question2026-11-11 16:10
Winnow-E4B, three wordingsbigger chat models12 hard jobs (other), not drawn–−2–droppedGemma Q8, three verdict wordings pooled2026-11-11 16:22
Wald-4B, three wordingsbigger chat models12 hard jobs (other), not drawn–−2–droppedGemma Q8, three verdict wordings pooled2026-11-11 17:32
Gemma-26B-A4B, production wordingbigger chat modelshalf B32+3product +1, test 0, infra +2dropped (coverage: 3 of 35 jobs timed out, so it was scored on 32 only and not counted as a pass)production prompt, Gemma Q82026-11-11 20:27
Gemma-26B-A4B, definitionsbigger chat modelshalf B32+2product +1, test −1, infra +2droppedfuller class definitions, Gemma Q82026-11-11 20:27
Gemma-26B-A4B, with hintbigger chat modelshalf B32−4product −2, test −3, infra +1droppeddecision model's estimate as a hint, Gemma Q82026-11-11 20:27
Gemma-26B-A4B, production wordingbigger chat modelsboth halves (A+B)68+15product +4, test +4, infra +7dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass)production prompt, Gemma Q82026-11-11 20:28
Gemma-26B-A4B, definitionsbigger chat modelsboth halves (A+B)68+7product +5, test −2, infra +4dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass)fuller class definitions, Gemma Q82026-11-11 20:28
Gemma-26B-A4B, with hintbigger chat modelsboth halves (A+B)680product 0, test −3, infra +3droppeddecision model's estimate as a hint, Gemma Q82026-11-11 20:28
Gemma-26B-A4B, production wordingbigger chat modelsboth halves (A+B)68−1product +3, test −2, infra −2droppeddecision model with the environment question2026-11-11 20:28
Gemma-26B-A4B, production wordingbigger chat modelshalf B32−3product 0, test −2, infra −1droppeddecision model with the environment question2026-11-11 20:28
Kev-9B, environment ruleother decision modelshalf A36−10product −8, test −5, infra +3droppeddecision model with the environment question2026-11-10 17:36
Kev-9B, label questionother decision modelshalf A36+3product −1, test 0, infra +4passeddecision model, label question only2026-11-10 17:36
Kev-4B, environment ruleother decision modelshalf A36−8product −6, test −5, infra +3droppeddecision model with the environment question2026-11-10 17:51
Kev-4B, label questionother decision modelshalf A36−7product −8, test −5, infra +6droppeddecision model, label question only2026-11-10 17:51
Clef-Flash, environment ruleother decision modelshalf A36−3product −4, test +4, infra −3droppeddecision model with the environment question2026-11-10 18:12
Clef-Flash, label questionother decision modelshalf A36−2product −5, test +3, infra 0droppeddecision model, label question only2026-11-10 18:12
d3-flash, environment ruleother decision modelshalf A36−3product −5, test +4, infra −2droppeddecision model with the environment question2026-11-10 19:00
Wald-4B as a decision modelother decision modelshalf A, not drawn00product 0, test 0, infra 0droppeddecision model with the environment question2026-11-10 19:00
d3-flash, label questionother decision modelshalf A36−2product −5, test +4, infra −1droppeddecision model, label question only2026-11-10 19:01
Kev-9B, label questionother decision modelshalf B35−2product −4, test +1, infra +1droppeddecision model, label question only2026-11-10 19:47
Kev-9B, environment ruleother decision modelshalf B35−7product −9, test −4, infra +6droppeddecision model with the environment question2026-11-10 19:47
Kev-9B, label questionother decision modelshalf B35−5product −5, test +1, infra −1droppeddecision model with the environment question2026-11-10 19:47
Kev-9B, label questionother decision modelsboth halves (A+B)71+1product −5, test +1, infra +5droppeddecision model, label question only2026-11-10 19:47
Kev-27B, environment ruleother decision modelshalf A36+2product +1, test 0, infra +1droppeddecision model with the environment question2026-11-11 05:41
Kev-27B, label questionother decision modelshalf A36+5product +1, test 0, infra +4passeddecision model, label question only2026-11-11 05:41
Kev-27B, label questionother decision modelshalf B35+4product −2, test 0, infra +6dropped (class rule: product down by 2 (the rule allows 1 on A or B))decision model, label question only2026-11-11 07:00
Kev-27B, environment ruleother decision modelshalf B35+1product −3, test 0, infra +4droppeddecision model with the environment question2026-11-11 07:00
Kev-27B, label questionother decision modelsboth halves (A+B)71+9product −1, test 0, infra +10passeddecision model, label question only2026-11-11 07:00
Kev-27B, environment ruleother decision modelsboth halves (A+B)71+3product −2, test 0, infra +5droppeddecision model with the environment question2026-11-11 07:00
hint plus environment overridecombination ruleshalf B33+2product +1, test +1, infra 0droppeddecision model with the environment question2026-11-10 11:37
hint plus environment overridecombination ruleshalf A33+1product +1, test 0, infra 0droppeddecision model with the environment question2026-11-10 11:37
nearest labelled precedent rulecombination ruleshalf A36−1product 0, test −1, infra 0droppeddecision model with the environment question2026-11-10 11:41
nearest labelled precedent rulecombination ruleshalf B350product −1, test +1, infra 0droppeddecision model with the environment question2026-11-10 11:41
investigation clone resultscombination rulesno screen run, not drawn–––dropped (coverage)–2026-11-10 11:44
hint or definitions plus environment overridecombination rulesboth halves (A+B), not drawn–––dropped–2026-11-10 11:44
yes/no overridescombination rulesno screen run, not drawn–––dropped–2026-11-10 11:44
six new yes/no questionscombination ruleshalf A, not drawn–––dropped–2026-11-10 14:01
Kev-27B and elephant-1 averagedcombination ruleshalf A36−1product −1, test −1, infra +1droppeddecision model with the environment question2026-11-11 05:43
Kev-27B and elephant-1 averagedcombination ruleshalf B35+3product −1, test 0, infra +4passeddecision model with the environment question2026-11-11 07:00
Kev-27B and elephant-1 averagedcombination rulesboth halves (A+B)71+2product −2, test −1, infra +5droppeddecision model with the environment question2026-11-11 07:00
verdict asked againsampling and votinghalf A360product +1, test −1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 15:02
verdict in a fresh conversationsampling and votinghalf A36+3product 0, test +3, infra 0passedfuller class definitions, Gemma Q82026-11-10 15:02
verdict in a fresh conversationsampling and votinghalf A36−4product −1, test +1, infra −4droppeddecision model with the environment question2026-11-10 15:02
verdict in a fresh conversationsampling and votingboth halves (A+B)71+3product −1, test +4, infra 0droppedfuller class definitions, Gemma Q82026-11-10 15:41
verdict in a fresh conversationsampling and votinghalf B350product −1, test +1, infra 0droppedfuller class definitions, Gemma Q82026-11-10 15:41
majority of five samplessampling and votinghalf A36+1product +1, test 0, infra 0droppedfuller class definitions, Gemma Q82026-11-10 16:30
flag: decision model disagrees with Gemmaflagshalf A, not drawn–––dropped–2026-11-10 13:14
flag: the five samples disagreeflagshalf A, not drawn–––dropped–2026-11-10 16:31
flag: top vote 3 of 5 or lessflagshalf B, not drawn–––dropped–2026-11-10 17:20
TF-IDF baselineclassic baselinehalf A36+9product +2, test +4, infra +3passed (passed A, then a leakage check showed it had learnt to recognise tickets (jobs of one ticket on both sides of its split); trained only on earlier jobs it was level with the decision rule)decision model with the environment question2026-11-10 11:34
TF-IDF leakage checkclassic baselineboth halves (A+B), not drawn–––dropped–2026-11-10 11:44

Bigger models did better than Gemma, but no better than the rule in the next section. On the evening of 11 November I stopped the slow models. I felt we were not investing our time most efficiently.

A second model

Gemma is a chat model: it writes text, and the script parses a class out of it. A decision model takes a question and a list of allowed answers, and returns a probability for each answer. I used elephant-1, a 9B decision model that runs on the same llama.cpp.

One writes, one weighs A chat model writes an answer token by token; a decision model returns a probability for each answer it was given. Chat model writes Gemma 4 E4B prompt with the job state generates token by token thinking, about 90% of the tokens text … Classification: Product bug the class is parsed from the last lines probability of the answer token product bug 1.0 about 1.0 whatever the answer, so it says nothing about confidence Decision model weighs elephant-1, 9B job state + question + list of answers returns one probability per answer product bug 0.43 test issue 0.35 infrastructure 0.22 second question, yes or no did the test environment fail? yes 0.06 threshold 0.3 Rule used in the article: an environment answer of at least 0.3 means infrastructure, otherwise whichever of product bug and test issue has the higher probability. Here 0.06 is below 0.3, so: product bug.
One writes, one weighs A chat model writes an answer token by token; a decision model returns a probability for each answer it was given. Chat model writes Gemma 4 E4B prompt with the job state generates token by token thinking, about 90% of the tokens text … Classification: Product bug the class is parsed from the last lines probability of the answer token product bug 1.0 about 1.0 whatever the answer, so it says nothing about confidence Decision model weighs elephant-1, 9B job state + question + list of answers returns one probability per answer product bug 0.43 test issue 0.35 infrastructure 0.22 second question, yes or no did the test environment fail? yes 0.06 threshold 0.3 Rule used in the article: an environment answer of at least 0.3 means infrastructure, otherwise whichever of product bug and test issue has the higher probability. Here 0.06 is below 0.3, so: product bug.
Figure 8. Left: a chat model writes text, and the class is read from its last lines. Right: a decision model is given the possible answers and returns a probability for each.
Data
  1. Chat model, Gemma 4 E4B:
    1. In: the prompt with the job state.
    2. Out: a block of thinking tokens, about 90% of what it generates, then text ending with “Classification: Product bug”.
    3. The class is parsed from the last lines.
    4. The probability of the answer token after all that thinking is about 1.0 whatever the answer, so it says nothing about confidence.
  2. Decision model, elephant-1 (9B):
    1. In: the same job state, a question and a fixed list of answers.
    2. Out: one probability per answer, for example product bug 0.43, test issue 0.35, infrastructure 0.22.
    3. A second yes/no question, “did the test environment fail?”, gets 0.06 against a threshold of 0.3.
  3. Rule used in the article: an environment answer of at least 0.3 means infrastructure, otherwise whichever of product bug and test issue has the higher probability. Here 0.06 is below 0.3, so the answer is product bug.

On its own it was no better than Gemma: on the gold jobs it was right on 42 of 95 against Gemma’s 50. But one question picked out infrastructure. Asked “did the test environment fail?” on the 116 jobs I tuned on, it gave “yes” a probability of 0.3 or more on 19 of the 30 infrastructure jobs and on 5 of the 86 others. I read that threshold off the data after seeing it.

Probability given to “the test environment failed” for 128 jobs, by hindsight label On 128 jobs the threshold was not read from, 22 of 42 infrastructure jobs get 0.3 or more for “the test environment failed”, against 2 of 70 product bugs and 3 of 16 test issues; the threshold was chosen after seeing other jobs. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 probability given to “the test environment failed” product bug, 0.0–0.1: 28 jobs product bug, 0.1–0.2: 35 jobs product bug, 0.2–0.3: 5 jobs product bug, 0.3–0.4: 1 job product bug, 0.4–0.5: 1 job test issue, 0.0–0.1: 13 jobs test issue, 0.3–0.4: 2 jobs test issue, 0.4–0.5: 1 job infrastructure, 0.0–0.1: 6 jobs infrastructure, 0.1–0.2: 14 jobs infrastructure, 0.3–0.4: 1 job infrastructure, 0.4–0.5: 3 jobs infrastructure, 0.5–0.6: 5 jobs infrastructure, 0.6–0.7: 6 jobs infrastructure, 0.7–0.8: 2 jobs infrastructure, 0.8–0.9: 5 jobs threshold 0.3, chosen after seeing other jobs product bug 2 of 70 at or above 0.3 test issue 3 of 16 at or above 0.3 infrastructure 22 of 42 at or above 0.3 The threshold was read from 116 other jobs, internal ones included; on those, 19 of 30 infrastructure jobs and 5 of 86 others were at or above 0.3.
Probability given to “the test environment failed” for 128 jobs, by hindsight label On 128 jobs the threshold was not read from, 22 of 42 infrastructure jobs get 0.3 or more for “the test environment failed”, against 2 of 70 product bugs and 3 of 16 test issues; the threshold was chosen after seeing other jobs. 0.0 0.2 0.4 0.6 0.8 1.0 probability given to “the test environment failed” product bug, 0.0–0.1: 28 jobs product bug, 0.1–0.2: 35 jobs product bug, 0.2–0.3: 5 jobs product bug, 0.3–0.4: 1 job product bug, 0.4–0.5: 1 job test issue, 0.0–0.1: 13 jobs test issue, 0.3–0.4: 2 jobs test issue, 0.4–0.5: 1 job infrastructure, 0.0–0.1: 6 jobs infrastructure, 0.1–0.2: 14 jobs infrastructure, 0.3–0.4: 1 job infrastructure, 0.4–0.5: 3 jobs infrastructure, 0.5–0.6: 5 jobs infrastructure, 0.6–0.7: 6 jobs infrastructure, 0.7–0.8: 2 jobs infrastructure, 0.8–0.9: 5 jobs threshold 0.3, chosen after seeing other jobs product bug 2 of 70 at or above 0.3 test issue 3 of 16 at or above 0.3 infrastructure 22 of 42 at or above 0.3 The threshold was read from 116 other jobs, internal ones included; on those, 19 of 30 infrastructure jobs and 5 of 86 others were at or above 0.3.
Figure 9. Each dot is one of the 128 jobs of the test in the next section, placed by the probability elephant-1 gave to a failed test environment. Dots in the grey area, at 0.3 or more, are called infrastructure: 22 of the 42 infrastructure jobs, and 5 of the 86 others.
Data
Jobs whose probability for “the test environment failed” is at or above the threshold 0.3, which was chosen after seeing the 116 jobs in the last two rows
Hindsight labelJobsAt or above 0.3Below 0.3Total
product bug128 jobs the threshold was not read from26870
test issue128 jobs the threshold was not read from31316
infrastructure128 jobs the threshold was not read from222042
infrastructure116 jobs the threshold was read from, internal jobs included191130
product bug or test issue116 jobs the threshold was read from, internal jobs included58186
Jobs per probability bin on the 128 jobs the threshold was not read from (each bin includes its lower bound)
Probabilityproduct bugtest issueinfrastructure
0.0–0.128136
0.1–0.235014
0.2–0.3500
0.3–0.4121
0.4–0.5113
0.5–0.6005
0.6–0.7006
0.7–0.8002
0.8–0.9005
0.9–1.0000

That left a rule with two questions and no Gemma in it. If the environment answer is 0.3 or more, the class is infrastructure. Otherwise it is whichever of product and test elephant-1 rates higher. The package for its server is merged (openQA #8102); the change to the script is open (#814).

I also handed elephant-1’s answer to Gemma as a hint. That passed its screen on the small sample, but the steered Gemma repeated elephant-1’s label on 59 of 71 jobs, so the gain was elephant-1’s. What surprised me most was how one can steer an LLM to an incorrect conclusion.

The test and the audit

To test the rule I froze it first. I wrote the rule, the threshold and the scoring into a file, and then ran it on 128 o3 jobs of the silver set1. The comments posted in production had the class right on 66 of the 128. The rule had it right on 93.

Then I audited the labels. All 128 were silver labels. 108 of them had been made with hindsight, 36 of the 42 infrastructure labels among them. A rerun that passed is a fair sign of an infrastructure problem. It is not something a model can see in the log of the failed job. And the labeller had worked from the same written class definitions that I later gave to elephant-1.

Three checks do not depend on those labels. The plainest uses 35 gold jobs, labelled by people. The rule got 20 right. Production’s model and prompt, run again on the same input, got 19. These jobs were among the ones I had chosen the threshold on, which should have favoured the rule. The other two checks are in the lower half of the figure.

Jobs with the class right: posted comments against the rule On 128 jobs whose labels a model set, most of them with hindsight, the rule had the class right on 93 against 66 for the posted comments; on three checks that do not depend on those labels the lead is gone: 20 for the rule against 19 for a replay of production on 35 gold jobs, 16 against 20 saying infrastructure on the 50 that passed on a rerun, and 10 against 9 on the 14 with a bug reference from a reviewer. Posted comments Rule The 128 jobs, by class of the label Every label here was set by a model, most with hindsight. All jobs 128 jobs All jobs: posted comments right on 66 of 128 posted comments 66 All jobs: rule right on 93 of 128 rule 93 Product bug 70 jobs Product bug: posted comments right on 38 of 70 38 Product bug: rule right on 58 of 70 58 Test issue 16 jobs Test issue: posted comments right on 9 of 16 9 Test issue: rule right on 13 of 16 13 Infrastructure 42 jobs Infrastructure: posted comments right on 19 of 42 19 Infrastructure: rule right on 22 of 42 22 Three checks that do not depend on those labels The 14 are among the 128; the 35 are not, and the 50 only in part. Gold jobs: label set by a person 35 jobs Gold jobs: label set by a person: a replay of production right on 19 of 35 (product bug 13 of 19, test issue 6 of 16) replay of production 19 Gold jobs: label set by a person: rule right on 20 of 35 (product bug 14 of 19, test issue 6 of 16) 20 Passed on a rerun: said infrastructure 50 jobs Passed on a rerun: posted comments said infrastructure on 20 of 50; both said it on 12 20 Passed on a rerun: rule said infrastructure on 16 of 50; both said it on 12 16 Bug reference from a reviewer 14 jobs Bug reference from a reviewer: posted comments right on 9 of 14 (all product bugs, 2 bugs in all) 9 Bug reference from a reviewer: rule right on 10 of 14 (all product bugs, 2 bugs in all) 10 Posted comments Rule The 128 jobs, by where the label came from Every label here was set by a model. Label given by a pass that used hindsight: 108; without hindsight: 20. All jobs 128 jobs All jobs: posted comments right on 66 of 128 posted comments 66 All jobs: rule right on 93 of 128 rule 93 Label resting on a passing rerun 23 jobs Label resting on a passing rerun: posted comments right on 16 of 23 (test issue 2 of 2, infrastructure 14 of 21) 16 Label resting on a passing rerun: rule right on 13 of 23 (test issue 2 of 2, infrastructure 11 of 21) 13 Other hindsight label 85 jobs Other hindsight label: posted comments right on 40 of 85 (product bug 32 of 60, test issue 5 of 10, infrastructure 3 of 15) 40 Other hindsight label: rule right on 68 of 85 (product bug 50 of 60, test issue 10 of 10, infrastructure 8 of 15) 68 Label made without hindsight 20 jobs Label made without hindsight: posted comments right on 10 of 20 (product bug 6 of 10, test issue 2 of 4, infrastructure 2 of 6) 10 Label made without hindsight: rule right on 12 of 20 (product bug 8 of 10, test issue 1 of 4, infrastructure 3 of 6) 12 Three checks that do not depend on those labels The 14 are among the 128; the 35 are not, and the 50 only in part. Gold jobs: label set by a person 35 jobs Gold jobs: label set by a person: a replay of production right on 19 of 35 (product bug 13 of 19, test issue 6 of 16) replay of production 19 Gold jobs: label set by a person: rule right on 20 of 35 (product bug 14 of 19, test issue 6 of 16) 20 Passed on a rerun: said infrastructure 50 jobs Passed on a rerun: posted comments said infrastructure on 20 of 50; both said it on 12 20 Passed on a rerun: rule said infrastructure on 16 of 50; both said it on 12 16 Bug reference from a reviewer 14 jobs Bug reference from a reviewer: posted comments right on 9 of 14 (all product bugs, 2 bugs in all) 9 Bug reference from a reviewer: rule right on 10 of 14 (all product bugs, 2 bugs in all) 10
Jobs with the class right: posted comments against the rule On 128 jobs whose labels a model set, most of them with hindsight, the rule had the class right on 93 against 66 for the posted comments; on three checks that do not depend on those labels the lead is gone: 20 for the rule against 19 for a replay of production on 35 gold jobs, 16 against 20 saying infrastructure on the 50 that passed on a rerun, and 10 against 9 on the 14 with a bug reference from a reviewer. Posted comments Rule The 128 jobs, by class of the label Every label here was set by a model, most with hindsight. All jobs 128 jobs All jobs: posted comments right on 66 of 128 posted comments 66 All jobs: rule right on 93 of 128 rule 93 Product bug 70 jobs Product bug: posted comments right on 38 of 70 38 Product bug: rule right on 58 of 70 58 Test issue 16 jobs Test issue: posted comments right on 9 of 16 9 Test issue: rule right on 13 of 16 13 Infrastructure 42 jobs Infrastructure: posted comments right on 19 of 42 19 Infrastructure: rule right on 22 of 42 22 Three checks that do not depend on those labels The 14 are among the 128; the 35 are not, and the 50 only in part. Gold jobs: label set by a person 35 jobs Gold jobs: label set by a person: a replay of production right on 19 of 35 (product bug 13 of 19, test issue 6 of 16) replay of production 19 Gold jobs: label set by a person: rule right on 20 of 35 (product bug 14 of 19, test issue 6 of 16) 20 Passed on a rerun: said infrastructure 50 jobs Passed on a rerun: posted comments said infrastructure on 20 of 50; both said it on 12 20 Passed on a rerun: rule said infrastructure on 16 of 50; both said it on 12 16 Bug reference from a reviewer 14 jobs Bug reference from a reviewer: posted comments right on 9 of 14 (all product bugs, 2 bugs in all) 9 Bug reference from a reviewer: rule right on 10 of 14 (all product bugs, 2 bugs in all) 10 Posted comments Rule The 128 jobs, by where the label came from Every label here was set by a model. Label given by a pass that used hindsight: 108; without hindsight: 20. All jobs 128 jobs All jobs: posted comments right on 66 of 128 posted comments 66 All jobs: rule right on 93 of 128 rule 93 Label resting on a passing rerun 23 jobs Label resting on a passing rerun: posted comments right on 16 of 23 (test issue 2 of 2, infrastructure 14 of 21) 16 Label resting on a passing rerun: rule right on 13 of 23 (test issue 2 of 2, infrastructure 11 of 21) 13 Other hindsight label 85 jobs Other hindsight label: posted comments right on 40 of 85 (product bug 32 of 60, test issue 5 of 10, infrastructure 3 of 15) 40 Other hindsight label: rule right on 68 of 85 (product bug 50 of 60, test issue 10 of 10, infrastructure 8 of 15) 68 Label made without hindsight 20 jobs Label made without hindsight: posted comments right on 10 of 20 (product bug 6 of 10, test issue 2 of 4, infrastructure 2 of 6) 10 Label made without hindsight: rule right on 12 of 20 (product bug 8 of 10, test issue 1 of 4, infrastructure 3 of 6) 12 Three checks that do not depend on those labels The 14 are among the 128; the 35 are not, and the 50 only in part. Gold jobs: label set by a person 35 jobs Gold jobs: label set by a person: a replay of production right on 19 of 35 (product bug 13 of 19, test issue 6 of 16) replay of production 19 Gold jobs: label set by a person: rule right on 20 of 35 (product bug 14 of 19, test issue 6 of 16) 20 Passed on a rerun: said infrastructure 50 jobs Passed on a rerun: posted comments said infrastructure on 20 of 50; both said it on 12 20 Passed on a rerun: rule said infrastructure on 16 of 50; both said it on 12 16 Bug reference from a reviewer 14 jobs Bug reference from a reviewer: posted comments right on 9 of 14 (all product bugs, 2 bugs in all) 9 Bug reference from a reviewer: rule right on 10 of 14 (all product bugs, 2 bugs in all) 10
Figure 10. Grey: the posted comments (for the 35 gold jobs, production’s model and prompt run again). Purple: the rule. Top: how many jobs each had right against the labels a model made; the rule is far ahead, 93 to 66. Bottom: the three checks that do not use those labels. For the 50 jobs that passed when rerun, the bars count how often each said infrastructure. In these three checks the rule is one job ahead twice and four jobs behind once.
Data
Jobs with the class right, posted comments against the rule. The rows from the first down to those under 'Label made without hindsight' are the 128 public jobs whose reference label a model set; a label counts as hindsight when the labelling pass that gave it set the hindsight flag: 108 jobs, the rows 'Label resting on a passing rerun' and 'Other hindsight label'; with 'Label made without hindsight' (20) they split the 128. The last rows are the three checks that do not depend on those labels: a person or a passing rerun decided the class. On the 35 gold jobs the comparison is a replay of production, not the posted comments. The 14 jobs with a bug reference from a reviewer are among the 128 and take their class from it; the hindsight labeller could read those references, so their label is not independent of them, and two bugs say nothing about test issues or infrastructure.
SetJobsPosted commentsRule
All 128 jobs (label set by a model)1286693
All jobs: product bug703858
All jobs: test issue16913
All jobs: infrastructure421922
Label resting on a passing rerun231613
Label resting on a passing rerun: test issue22 of 22 of 2
Label resting on a passing rerun: infrastructure2114 of 2111 of 21
Other hindsight label854068
Other hindsight label: product bug6032 of 6050 of 60
Other hindsight label: test issue105 of 1010 of 10
Other hindsight label: infrastructure153 of 158 of 15
Label made without hindsight201012
Label made without hindsight: product bug106 of 108 of 10
Label made without hindsight: test issue42 of 41 of 4
Label made without hindsight: infrastructure62 of 63 of 6
Gold jobs: label set by a person (not among the 128; among the jobs the threshold was chosen on)3519 (replay of production)20
Gold jobs: product bug1913 of 19 (replay)14 of 19
Gold jobs: test issue166 of 16 (replay)6 of 16
Passed on a rerun (only partly among the 128): said infrastructure502016 (both said it: 12)
Bug reference from a reviewer (among the 128; 2 bugs in all, one on 12 jobs; all product bugs)14910

So the rule agrees much better than production with a hindsight labeller that shares its vocabulary, and no better with people.

The rerun

If an infrastructure label often means that a later run passed, then the best infrastructure detector on offer is the rerun. Of the 299 silver jobs, 188 were first failures: the previous run of the same scenario had not failed. 67 of those passed when they were run again in the same build2.

The hook starts investigation jobs only on BISECT: YES, and it retried 8 of the 67. The same comment also said whether to restart the job: Restart: YES on 28 of the 67. No code acted on that line. #821 starts one retry job when the comment says Restart: YES.

Commented jobs by what a rerun did, and the jobs each retry rule retries Of 299 commented jobs, 67 passed when simply run again in the same build; the hook's investigation retries covered 8 of them, while the comment said Restart: YES on 28. What became of the 299 commented jobs Commented jobs → First failures: 188 Commented jobs → Not a first failure: 111 First failures → Passed on a rerun: 67 First failures → Rerun, failed again: 44 First failures → Never rerun: 77 Commented jobs: 299 First failures: 188 Not a first failure: 111 Passed on a rerun: 67 Rerun, failed again: 44 Never rerun: 77 Commented jobs 299 Not a first failure 111 Passed on a rerun 67 Rerun, failed again 44 Never rerun 77 First failures 188 What each retry rule retries among the 299 commented jobs, and how many of the 67 it covers jobs it retries of the 67 it covers 0 50 100 150 200 jobs The hook as it was (its investigation retries): retried 35 jobs, 8 of them among the 67 that passed on a rerun 35 8 Restart: YES: would retry 62 jobs, 28 of them among the 67 that passed on a rerun 62 28 Restart: YES or BISECT: YES: would retry 105 jobs, 44 of them among the 67 that passed on a rerun 105 44 Every first failure: would retry 188 jobs, 67 of them among the 67 that passed on a rerun 188 67 The hook as it was (its investigation retries) Restart: YES Restart: YES or BISECT: YES Every first failure
Commented jobs by what a rerun did, and the jobs each retry rule retries Of 299 commented jobs, 67 passed when simply run again in the same build; the hook's investigation retries covered 8 of them, while the comment said Restart: YES on 28. What became of the 299 commented jobs Commented jobs → First failures: 188 Commented jobs → Not a first failure: 111 First failures → Passed on a rerun: 67 First failures → Rerun, failed again: 44 First failures → Never rerun: 77 Commented jobs: 299 First failures: 188 Not a first failure: 111 Passed on a rerun: 67 Rerun, failed again: 44 Never rerun: 77 Commented jobs 299 Passed on a rerun 67 Failed again 44 Never rerun 77 First failures 188 Not a first failure 111 What each retry rule retries among the 299 commented jobs, and how many of the 67 it covers jobs it retries of the 67 it covers 0 50 100 150 200 jobs The hook as it was (its investigation retries): retried 35 jobs, 8 of them among the 67 that passed on a rerun 35 8 Restart: YES: would retry 62 jobs, 28 of them among the 67 that passed on a rerun 62 28 Restart: YES or BISECT: YES: would retry 105 jobs, 44 of them among the 67 that passed on a rerun 105 44 Every first failure: would retry 188 jobs, 67 of them among the 67 that passed on a rerun 188 67 The hook as it was (its investigation retries) Restart: YES Restart: YES or BISECT: YES Every first failure
Figure 11. Top: what became of the 299 jobs. 188 were first failures, and 67 of those passed when run again. Bottom: four possible retry rules. Grey is how many jobs a rule retries, purple how many of the 67 it covers.
Data
What became of 299 commented jobs: 67 of 188 first failures passed when run again in the same build
JobsCount
Commented jobs299
First failures188
Not a first failure111
First failures: passed on a rerun67
First failures: rerun, failed again44
First failures: never rerun77
Jobs each retry rule retries among the 299 commented jobs (the hook's are actual retries, the others what the rule would have retried) and how many of the 67 passing reruns each covers
Retry ruleJobs it retriesOf the 67 that passed on a rerun, it covers
The hook as it was (its investigation retries)358
Restart: YES6228
Restart: YES or BISECT: YES10544
Every first failure18867

Pull requests

I opened 28 pull requests against os-autoinst-scripts and openQA between 25 October and 13 November. 22 are merged. The work is tracked in a public ticket.

topicmergedopen
Prompt and verdict rules#789 #790 #791 #795 #798 #801 #802
Comments and the hook#788 #796 #797 openQA #8023#817 #819 openQA #8113
Prompt length#799 #800#815
Routing and the model server#808 openQA #8069 openQA #8070 openQA #8071
Groups nobody reviews#810
Decision modelopenQA #8102#814
Retry on Restart: YES#821
CI#792 #809 #816

What I would do differently

  • Count the labels before the first experiment, per class and per ticket. On day one that would have shown the 30-point limit.
  • Measure the noise floor next and size every pass rule from it.
  • Ask of every label where it comes from.
  • Read every field the system already produces before building anything to replace it.

How this was done

I wrote the scripts and ran the experiments with an AI coding assistant. A frontier model made the silver labels, checked the gold tickets that did not map cleanly, and was the larger model in the comparison on Gemma’s input.

The data comes from o3, which is public, but job pages there expire, so I link none. Parts of several sets came from an internal instance and appear as counts only. The numbers behind every figure are in the repository of this site.

As for the comments themselves, they seem useful. I hope the open pull requests get merged.


  1. None of the 128 was among the jobs the threshold had been read from, though I had partly worded the question on the same pool, and 12 share a test and a build with jobs I tuned on. The file named two versions: as the main rule, one that asks the product-or-test question with the options in three orders, and this two-question rule as the cheaper fallback. Both met the bar written in the file, with 95 and 93, and I report the cheaper one. ↩︎

  2. The comments on the pull requests say 68. One of those reruns had failed and was set to passed by a force_result label after a retry of it passed. ↩︎