On openqa.opensuse.org, a small language model comments on failed test jobs. It gets what a human reviewer would look at and says what kind of failure it sees. In the autumn of 2026 I tuned it. The first pull requests are from 25 October, and most of the work was done in the week of 7 to 14 November.
I started mostly out of curiosity, to see how much I could improve with constrained resources. The constraint was the small model. I had no particular goal beyond exploration and perhaps some improvement.
The short version:
- Small fixes to the prompt and the script made visible differences.
- Whether the model’s verdict is right could hardly be measured, because too few failures had a cause recorded by a person.
- One change looked like a clear win until I checked where the right answers it was scored against came from.
- What looked most useful was a plain rerun. The comment already said whether to restart a job, and no code acted on that line.
What openQA does and what the model adds
openQA tests an operating system the way a person would. It boots a machine, types, clicks, and compares the screen with reference screenshots, called needles, and the serial console with expected text. One run of one test scenario on one build of the system is a job. openqa.opensuse.org, o3 for short, is the public instance that tests openSUSE.
When a job fails, somebody has to decide what kind of failure it is. There are three kinds:
- a product bug: the software under test is wrong;
- a test issue: the test is wrong or out of date;
- infrastructure: a worker (the machine that runs the job), the network or some outside server let the job down.
A reviewer records the decision by linking a bug or a ticket on the job. That link is called a bug reference.
o3 also asks a language model. When a job fails and no known issue matches it, a hook on o3 runs a script
called openqa-llm-investigate. The script sends what a reviewer would look at to Gemma 4 E4B. The model
runs under llama.cpp on three of the worker hosts, without a graphics card, in a heavily compressed form
(quantised to a level called IQ2_M). Its reply becomes a comment on the job:
The three lines before the footer are meant for programs: the class, whether to bisect, and whether to
restart the job. Only BISECT: YES starts anything: the hook then schedules investigation jobs that retry
the test and search for the change that broke it.
BISECT: YES, and nothing is connected to Restart.Data
- A test job fails on openQA. If no known issue matches, the job-done hook runs openqa-llm-investigate.
- The script builds one prompt from:
- the failure analysis of the job's log, with earlier reviews of runs failing in the same module
- the settings difference to the last good job
- the test commits since the last good job
- It asks a llama.cpp server running Gemma 4 E4B (three hosts behind nginx) three questions in one chat:
- question 1: the analysis, one line per failed module
- question 2: a summary of at most two sentences
- question 3: the verdict, with the job statistics
- The script posts a comment on the job that ends with three lines:
- Classification: Product bug | Test issue | Infrastructure problem. Only people read it; they reviewed 2.2% of the commented jobs.
- BISECT: YES | NO. On YES the hook starts openqa-investigate (a retry and reruns with the last good tests and build) and then openqa-trigger-bisect-jobs. This is the only answer wired to anything.
- Restart: YES | NO. Not connected: nothing acts on it.
In 42 days the model commented on 6459 jobs in four job groups. A person later reviewed about 140 of them, 2.2%.
The easy fixes
The first fixes needed no statistics.
The prompt, the text that tells the model what to do, carried a hint about outages that fired on nearly every build. On 17 jobs I audited by hand, the model called 15 an infrastructure problem. After the prompt required evidence for that verdict, 3 of 17 were called infrastructure, and the class matched my audit on 12 of 17 jobs, up from 4 (#802).
Half of the commented jobs, 3290 or 51%, were in the Development group, where a person reviewed 4. The script now skips that group (#810).
The third fix is about length. A model reads text as tokens, which are pieces of words, and can take in only so many. The script can cut a long prompt to fit such a limit, if one is set; as far as I can see, o3 sets none. It cuts by characters and assumed 2.5 characters per token (#799). Most real prompts have fewer, so a cut prompt can still be too long. I proposed 1.9 (#815).
Data
| Characters per token | Prompts |
|---|---|
| 1.70–1.75 | 1 |
| 1.75–1.80 | 0 |
| 1.80–1.85 | 0 |
| 1.85–1.90 | 0 |
| 1.90–1.95 | 1 |
| 1.95–2.00 | 3 |
| 2.00–2.05 | 18 |
| 2.05–2.10 | 11 |
| 2.10–2.15 | 29 |
| 2.15–2.20 | 65 |
| 2.20–2.25 | 79 |
| 2.25–2.30 | 92 |
| 2.30–2.35 | 57 |
| 2.35–2.40 | 36 |
| 2.40–2.45 | 23 |
| 2.45–2.50 | 19 |
| 2.50–2.55 | 23 |
| 2.55–2.60 | 15 |
| 2.60–2.65 | 5 |
| 2.65–2.70 | 4 |
| 2.70–2.75 | 4 |
| 2.75–2.80 | 8 |
| 2.80–2.85 | 1 |
| 2.85–2.90 | 2 |
| 2.90–2.95 | 0 |
| 2.95–3.00 | 0 |
| 3.00–3.05 | 0 |
| 3.05–3.10 | 1 |
| 3.10–3.15 | 1 |
| 3.15–3.20 | 1 |
| 3.20–3.25 | 1 |
| 3.25–3.30 | 1 |
| 3.30–3.35 | 2 |
| 3.35–3.40 | 0 |
| 3.40–3.45 | 3 |
| 3.45–3.50 | 1 |
| 3.50–3.55 | 2 |
| 3.55–3.60 | 4 |
| 3.60–3.65 | 1 |
| 3.65–3.70 | 2 |
| 3.70–3.75 | 1 |
| 3.75–3.80 | 1 |
| 3.80–3.85 | 0 |
| 3.85–3.90 | 1 |
| 3.90–3.95 | 1 |
| All prompts | 520 |
Too few labels
To say that a change makes the class right more often, I need failures whose class is known. Such a known class is a label.
Reviewers are the natural source. In 42 days, 1316 failed jobs carried a bug reference, but 1226 of them got it from a bot or by carry-over from an earlier job of the same scenario. 90 were set by a person, and of the 83 whose ticket is public, not one is an infrastructure problem. A network hiccup usually gets a restart, not a ticket.
Going back further gave 98 jobs with a reference set by a person, 50 product bugs and 48 test issues. This is the gold set. It comes from 27 tickets, since one bug can fail many jobs.
For infrastructure I had a frontier model, one of the most capable models available, label 299 failed o3 jobs. This is the silver set. No person checked these labels, and the labeller had hindsight, which the commenting model lacks: it could see whether the same scenario passed an hour later.
Data
| Source | Part | Jobs | Note |
|---|---|---|---|
| Bug references | failed jobs with a bug reference | 1,316 | 42 days |
| Bug references | set by a bot or carried over | 1,226 | |
| Bug references | set by a person | 90 | 23 distinct references |
| Set by a person | product bug | 41 | |
| Set by a person | test issue | 40 | |
| Set by a person | class unknown | 7 | ticket that is not public |
| Set by a person | ticket of another team | 2 | |
| Set by a person | infrastructure | 0 | none of the 83 with a public ticket |
| Gold set | all jobs | 98 | 27 tickets |
| Gold set | product bug | 50 | 14 tickets |
| Gold set | test issue | 48 | 13 tickets |
| Gold set | do not map cleanly onto the three classes | 40 | 8 tickets |
| Gold set | ticket 1 | 18 | test issue, does not map cleanly |
| Gold set | ticket 2 | 8 | product bug |
| Gold set | ticket 3 | 8 | test issue |
| Gold set | ticket 4 | 7 | product bug |
| Gold set | ticket 5 | 7 | product bug, does not map cleanly |
| Gold set | ticket 6 | 6 | product bug |
| Gold set | ticket 7 | 5 | test issue, does not map cleanly |
| Gold set | ticket 8 | 4 | product bug |
| Gold set | ticket 9 | 4 | product bug |
| Gold set | ticket 10 | 4 | test issue, does not map cleanly |
| Gold set | ticket 11 | 3 | product bug |
| Gold set | ticket 12 | 3 | test issue |
| Gold set | ticket 13 | 2 | product bug |
| Gold set | ticket 14 | 2 | product bug |
| Gold set | ticket 15 | 2 | product bug |
| Gold set | ticket 16 | 2 | product bug, does not map cleanly |
| Gold set | ticket 17 | 2 | test issue |
| Gold set | ticket 18 | 2 | test issue, does not map cleanly |
| Gold set | ticket 19 | 1 | product bug |
| Gold set | ticket 20 | 1 | product bug |
| Gold set | ticket 21 | 1 | product bug |
| Gold set | ticket 22 | 1 | test issue |
| Gold set | ticket 23 | 1 | test issue |
| Gold set | ticket 24 | 1 | test issue |
| Gold set | ticket 25 | 1 | test issue |
| Gold set | ticket 26 | 1 | test issue, does not map cleanly |
| Gold set | ticket 27 | 1 | test issue, does not map cleanly |
| Silver set | all jobs | 299 | labelled by a frontier model with hindsight |
| Silver set | product bug | 87 | 70 confident (labeller confidence at least 0.6) |
| Silver set | test issue | 87 | 26 confident (labeller confidence at least 0.6) |
| Silver set | infrastructure | 111 | 57 confident (labeller confidence at least 0.6) |
| Silver set | unclear | 14 |
Jobs that share a ticket are not independent, so what counts is the number of tickets. With 27, a comparison of two models can reliably show only a difference of about 30 percentage points. Fifteen points would need 105 tickets, and ten points 235.
Data
| Tickets | Smallest detectable difference (points) | Marked |
|---|---|---|
| 10 | 48.7 | |
| 15 | 39.8 | |
| 20 | 34.4 | |
| 25 | 30.8 | |
| 27 | 29.6 | what I had, about 30 |
| 30 | 28.1 | |
| 35 | 26.0 | |
| 40 | 24.3 | |
| 45 | 23.0 | |
| 50 | 21.8 | |
| 55 | 20.8 | |
| 60 | 19.9 | |
| 65 | 19.1 | |
| 70 | 18.4 | |
| 75 | 17.8 | |
| 80 | 17.2 | |
| 85 | 16.7 | |
| 90 | 16.2 | |
| 95 | 15.8 | |
| 100 | 15.4 | |
| 105 | 15.0 | about 15 |
| 110 | 14.7 | |
| 120 | 14.1 | |
| 130 | 13.5 | |
| 140 | 13.0 | |
| 150 | 12.6 | |
| 160 | 12.2 | |
| 170 | 11.8 | |
| 180 | 11.5 | |
| 190 | 11.2 | |
| 200 | 10.9 | |
| 210 | 10.6 | |
| 220 | 10.4 | |
| 230 | 10.2 | |
| 235 | 10.0 | about 10 |
| 240 | 9.9 | |
| 250 | 9.7 | |
| 260 | 9.6 | |
| 270 | 9.4 | |
| 280 | 9.2 | |
| 290 | 9.0 | |
| 300 | 8.9 |
A noisy model
Asked the same thing twice, the model does not always give the same class. For most experiments I used Q8, a less compressed version of the same Gemma. A second run a day later, with the same prompt and the same settings, kept the class on 29 of 35 gold jobs.
So if a new prompt gets three more jobs right than the old one, one run each, it has shown nothing.
Data
| Job set | Jobs | Kept its class | Changed |
|---|---|---|---|
| Gold jobs, run again a day later | 35 | 29 | 6 |
| Test issues among those gold jobs | 16 | 11 | 5 |
| Another 16 gold jobs (8 shared), run again four days later | 16 | 7 | 9 |
| Product bugs from an internal instance (internal counts) | 34 | 33 | 1 |
| Gold jobs and internal product bugs together (mixed counts; context, not drawn) | 69 | 62 | 7 |
| Verdict only, asked again on the same stored answers to the first two questions (mixed counts; context, not drawn) | 36 | 33 | 3 |
What the model gets wrong
Gemma leans towards product bug. Of the 46 test-issue jobs in the gold set that it answered, Gemma at Q8 called 39 a product bug and 6 a test issue. The IQ2_M version that runs in production called 15 of the 46 a test issue, but called only 35 of the 49 product bugs a product bug. Both got 50 of 95 jobs right.
Data
| Answers | Failure kind | Label | Answered product bug | Answered test issue | Answered infrastructure | Jobs |
|---|---|---|---|---|---|---|
| Gemma at Q8 | all jobs | product bug | 44 | 5 | 0 | 49 |
| Gemma at Q8 | all jobs | test issue | 39 | 6 | 1 | 46 |
| Gemma at Q8 | failed in a script | product bug | 22 | 0 | 0 | 22 |
| Gemma at Q8 | failed in a script | test issue | 13 | 0 | 0 | 13 |
| Gemma at Q8 | needle mismatch | product bug | 15 | 5 | 0 | 20 |
| Gemma at Q8 | needle mismatch | test issue | 18 | 6 | 0 | 24 |
| Gemma at Q8 | serial console | product bug | 3 | 0 | 0 | 3 |
| Gemma at Q8 | serial console | test issue | 6 | 0 | 1 | 7 |
| Gemma at Q8 | other | product bug | 4 | 0 | 0 | 4 |
| Gemma at Q8 | other | test issue | 2 | 0 | 0 | 2 |
| IQ2_M | all jobs | product bug | 35 | 13 | 1 | 49 |
| IQ2_M | all jobs | test issue | 31 | 15 | 0 | 46 |
| IQ2_M | failed in a script | product bug | 20 | 2 | 0 | 22 |
| IQ2_M | failed in a script | test issue | 8 | 5 | 0 | 13 |
| IQ2_M | needle mismatch | product bug | 9 | 11 | 0 | 20 |
| IQ2_M | needle mismatch | test issue | 15 | 9 | 0 | 24 |
| IQ2_M | serial console | product bug | 2 | 0 | 1 | 3 |
| IQ2_M | serial console | test issue | 6 | 1 | 0 | 7 |
| IQ2_M | other | product bug | 4 | 0 | 0 | 4 |
| IQ2_M | other | test issue | 2 | 0 | 0 | 2 |
| Silver, posted | all jobs | infrastructure | 33 | 45 | 33 | 111 |
| Silver, posted | failed in a script | infrastructure | 14 | 17 | 22 | 53 |
| Silver, posted | needle mismatch | infrastructure | 7 | 20 | 2 | 29 |
| Silver, posted | other | infrastructure | 12 | 8 | 9 | 29 |
A much larger model did no better that I could see. A frontier model given exactly Gemma’s input, with the written class definitions, was right on 65% of 26 gold tickets, averaged per ticket, against 60% for Gemma as production runs it.
What I tried
One run of ten prompt variants over the 116 labelled jobs I tuned prompts on took seven and a half hours, so I screened ideas on a small sample instead: 71 jobs with the three classes in equal parts, split into halves A and B. 29 of the 71 came from an internal instance, so for this set I give counts only.
The pass rule was written down before each run: at least three more jobs right than the comparison run, and no class losing two. An idea ran on A first, and on B only if it passed.
In two days I logged 179 comparisons: prompt wordings, added facts, sampling with a vote, other models. 31 passed and 148 did not. Of the 19 that passed on A and were then run on B, 18 failed there. That is what you see when the pass rule sits inside the run-to-run noise.
Data
| Idea | Family | Screen | Jobs | Net | Change per class | Verdict | Control | Logged |
|---|---|---|---|---|---|---|---|---|
| definitions per kind of failure | prompt wording | half A | 36 | +5 | product −1, test +1, infra +5 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 15:02 |
| quote the evidence | prompt wording | half A | 36 | 0 | product 0, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 15:02 |
| definitions per kind of failure | prompt wording | half B | 35 | +2 | product −3, test +1, infra +4 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 15:41 |
| definitions per kind of failure | prompt wording | half B | 35 | −3 | product −4, test 0, infra +1 | dropped | decision model with the environment question | 2026-11-10 15:41 |
| production wording | prompt wording | half A | 36 | −3 | product +1, test −2, infra −2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| production wording | prompt wording | half B | 35 | −1 | product 0, test −1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| no example | prompt wording | half A | 36 | −1 | product +1, test −2, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| no example | prompt wording | half B | 35 | +1 | product +1, test −1, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| step by step | prompt wording | half A | 36 | +3 | product +1, test +2, infra 0 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| step by step | prompt wording | half B | 35 | 0 | product −1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| few-shot examples | prompt wording | half A | 36 | +5 | product +4, test +1, infra 0 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| few-shot examples | prompt wording | half B | 35 | −1 | product −1, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| direct answer | prompt wording | half A | 36 | +1 | product +1, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| direct answer | prompt wording | half B | 35 | 0 | product −1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| may abstain | prompt wording | half A | 36 | −2 | product +1, test −2, infra −1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| may abstain | prompt wording | half B | 35 | −2 | product −3, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| challenge the first answer | prompt wording | half A | 36 | +2 | product +1, test 0, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| challenge the first answer | prompt wording | half B | 35 | +1 | product 0, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| contrast pairs in the first prompt | prompt wording | half A | 36 | +1 | product +1, test −1, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 21:05 |
| contrast pairs in the first prompt | prompt wording | half A | 36 | +4 | product +1, test +1, infra +2 | passed | production prompt, Gemma Q8 | 2026-11-10 21:05 |
| contrast pairs in the first prompt | prompt wording | half A | 36 | −6 | product 0, test −3, infra −3 | dropped | decision model with the environment question | 2026-11-10 21:05 |
| contrast pairs in the first prompt | prompt wording | half A | 36 | +1 | product +2, test −1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 21:05 |
| contrast pairs in the first prompt | prompt wording | both halves (A+B) | 71 | +4 | product +1, test +2, infra +1 | dropped | production prompt, Gemma Q8 | 2026-11-11 01:27 |
| contrast pairs in the first prompt | prompt wording | half B | 35 | 0 | product 0, test +1, infra −1 | dropped | production prompt, Gemma Q8 | 2026-11-11 01:27 |
| contrast pairs in the first prompt | prompt wording | both halves (A+B) | 71 | 0 | product +2, test −1, infra −1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 01:27 |
| contrast pairs in the first prompt | prompt wording | both halves (A+B) | 71 | 0 | product +1, test −2, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 01:27 |
| contrast pairs in the first prompt | prompt wording | both halves (A+B) | 71 | −12 | product 0, test −4, infra −8 | dropped | decision model with the environment question | 2026-11-11 01:27 |
| no note | prompt wording | half A | 36 | −1 | product +1, test −2, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| no note, abstentions filled | prompt wording | half A | 36 | −1 | product +1, test −2, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| no note, abstentions filled | prompt wording | half A | 36 | −8 | product 0, test −4, infra −4 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| reasoning first | prompt wording | half A | 36 | +3 | product 0, test +3, infra 0 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| reasoning first, abstentions filled | prompt wording | half A | 36 | +3 | product 0, test +3, infra 0 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| reasoning first, abstentions filled | prompt wording | half A | 36 | −4 | product −1, test +1, infra −4 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| name where it failed | prompt wording | half A | 36 | +2 | product +2, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| name where it failed, abstentions filled | prompt wording | half A | 36 | +2 | product +2, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| name where it failed, abstentions filled | prompt wording | half A | 36 | −5 | product +1, test −2, infra −4 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| proof required | prompt wording | half A | 36 | +3 | product −1, test +3, infra +1 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| proof required, abstentions filled | prompt wording | half A | 36 | +3 | product −1, test +3, infra +1 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| proof required, abstentions filled | prompt wording | half A | 36 | −4 | product −2, test +1, infra −3 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| debate | prompt wording | half A | 36 | −3 | product −1, test −3, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| debate, abstentions filled | prompt wording | half A | 36 | −3 | product −1, test −3, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| debate, abstentions filled | prompt wording | half A | 36 | −10 | product −2, test −5, infra −3 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| three views | prompt wording | half A | 36 | −2 | product −2, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| three views, abstentions filled | prompt wording | half A | 36 | 0 | product −1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| three views, abstentions filled | prompt wording | half A | 36 | −7 | product −2, test −1, infra −4 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| class probabilities | prompt wording | half A | 36 | +1 | product −3, test +4, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| class probabilities, abstentions filled | prompt wording | half A | 36 | +1 | product −3, test +4, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| class probabilities, abstentions filled | prompt wording | half A | 36 | −6 | product −4, test +2, infra −4 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| yes/no chain | prompt wording | half A | 36 | −5 | product −1, test −2, infra −2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| yes/no chain, abstentions filled | prompt wording | half A | 36 | −5 | product −1, test −2, infra −2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 05:04 |
| yes/no chain, abstentions filled | prompt wording | half A | 36 | −12 | product −2, test −4, infra −6 | dropped | decision model with the environment question | 2026-11-11 05:04 |
| reasoning first | prompt wording | half B | 35 | 0 | product −3, test +2, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| reasoning first | prompt wording | both halves (A+B) | 71 | +3 | product −3, test +5, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| reasoning first, abstentions filled | prompt wording | half B | 35 | 0 | product −3, test +2, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| reasoning first, abstentions filled | prompt wording | both halves (A+B) | 71 | −9 | product −5, test +2, infra −6 | dropped | decision model with the environment question | 2026-11-11 06:21 |
| proof required | prompt wording | half B | 35 | 0 | product −3, test +3, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| proof required | prompt wording | both halves (A+B) | 71 | +3 | product −4, test +6, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| proof required, abstentions filled | prompt wording | half B | 35 | 0 | product −3, test +3, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 06:21 |
| proof required, abstentions filled | prompt wording | both halves (A+B) | 71 | −9 | product −6, test +3, infra −6 | dropped | decision model with the environment question | 2026-11-11 06:21 |
| decision model on a focused excerpt | added facts | half A | 36 | −3 | product −1, test 0, infra −2 | dropped | decision model with the environment question | 2026-11-10 12:19 |
| decision model on Gemma's analysis only | added facts | half A | 36 | −8 | product +1, test −5, infra −4 | dropped | decision model with the environment question | 2026-11-10 12:42 |
| decision model on the excerpt plus Gemma's analysis | added facts | half A | 36 | 0 | product 0, test +2, infra −2 | dropped | decision model with the environment question | 2026-11-10 13:14 |
| job facts listed | added facts | half A | 36 | −1 | product +2, test −3, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| job facts listed | added facts | half B | 35 | −1 | product −2, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| decision model's estimate as a hint | added facts | half A | 36 | +2 | product +1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| decision model's estimate as a hint | added facts | half B | 35 | +4 | product +1, test +2, infra +1 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 19:01 |
| job statistics in the first prompt | added facts | half A | 36 | +1 | product +2, test −1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 21:43 |
| job statistics in the first prompt | added facts | half A | 36 | +1 | product +1, test 0, infra 0 | dropped | production prompt, Gemma Q8 | 2026-11-10 21:43 |
| job statistics in the first prompt | added facts | half A | 36 | −6 | product +1, test −3, infra −4 | dropped | decision model with the environment question | 2026-11-10 21:43 |
| job statistics in the first prompt | added facts | half A | 36 | −2 | product +2, test −2, infra −2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 21:43 |
| test source in the first prompt | added facts | half A | 36 | +4 | product +3, test −1, infra +2 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 22:39 |
| test source in the first prompt | added facts | half A | 36 | +4 | product +1, test +2, infra +1 | passed | production prompt, Gemma Q8 | 2026-11-10 22:39 |
| test source in the first prompt | added facts | half A | 36 | −3 | product +2, test −3, infra −2 | dropped | decision model with the environment question | 2026-11-10 22:39 |
| test source in the first prompt | added facts | half A | 36 | +1 | product +2, test 0, infra −1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 22:39 |
| decision model's area as a steer | added facts | half A | 36 | +2 | product +1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | +8 | product +1, test +4, infra +3 | passed | production prompt, Gemma Q8 | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | −5 | product 0, test −1, infra −4 | dropped | decision model with the environment question | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | +5 | product +2, test +2, infra +1 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | −2 | product +1, test 0, infra −3 | dropped | decision model with the environment question | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | +3 | product +1, test +1, infra +1 | passed | decision model's estimate as a hint, Gemma Q8 | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | half A | 36 | +4 | product 0, test +2, infra +2 | passed | test source in the prompt, production verdict | 2026-11-10 23:36 |
| decision model's area as a steer | added facts | both halves (A+B) | 71 | +11 | product +1, test +7, infra +3 | passed | production prompt, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | half B | 35 | +3 | product 0, test +3, infra 0 | passed | production prompt, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | half B | 35 | +2 | product 0, test +2, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | half B | 35 | −2 | product −1, test 0, infra −1 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | half B | 35 | −3 | product −1, test +1, infra −3 | dropped | decision model with the environment question | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | both halves (A+B) | 71 | −5 | product 0, test +1, infra −6 | dropped | decision model with the environment question | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | both halves (A+B) | 71 | +3 | product 0, test +2, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | half B | 35 | +1 | product −1, test +1, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 00:33 |
| decision model's area as a steer | added facts | both halves (A+B) | 71 | +1 | product 0, test +1, infra 0 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 00:33 |
| test source in the first prompt | added facts | half B | 35 | 0 | product 0, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 08:51 |
| test source in the first prompt | added facts | both halves (A+B) | 71 | +4 | product +3, test −1, infra +2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 08:51 |
| test source in the first prompt | added facts | half B | 35 | −1 | product +1, test 0, infra −2 | dropped | production prompt, Gemma Q8 | 2026-11-11 08:51 |
| test source in the first prompt | added facts | both halves (A+B) | 71 | +3 | product +2, test +2, infra −1 | dropped | production prompt, Gemma Q8 | 2026-11-11 08:51 |
| test source in the first prompt | added facts | both halves (A+B) | 71 | −8 | product +1, test −4, infra −5 | dropped | decision model with the environment question | 2026-11-11 08:51 |
| test source in the first prompt | added facts | both halves (A+B) | 71 | −2 | product +1, test −4, infra +1 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 08:51 |
| steer plus test source | added facts | half A | 36 | +5 | product 0, test +2, infra +3 | passed | production prompt, Gemma Q8 | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | −3 | product −1, test −2, infra 0 | dropped | steered analysis, production verdict | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | +2 | product +2, test −1, infra +1 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | 0 | product +1, test −2, infra +1 | dropped | steered analysis, definitions verdict | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | −5 | product 0, test −2, infra −3 | dropped | decision model with the environment question | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | −5 | product +1, test −3, infra −3 | dropped | decision model with the environment question | 2026-11-11 09:48 |
| steer plus test source | added facts | half A | 36 | +1 | product −1, test 0, infra +2 | dropped | test source in the prompt, production verdict | 2026-11-11 09:49 |
| steer plus test source | added facts | half A | 36 | −2 | product −1, test 0, infra −1 | dropped | test source in the prompt, definitions verdict | 2026-11-11 09:49 |
| Ornith-9B, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | +3 | – | dropped (cut at the 70-minute limit with 4 of 12 jobs answered (time rule)) | Gemma Q8, three verdict wordings pooled | 2026-11-11 10:58 |
| Gemma-26B-A4B, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | +8 | – | passed | Gemma Q8, three verdict wordings pooled | 2026-11-11 11:59 |
| Qwen3.5-35B, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | 0 | – | dropped | Gemma Q8, three verdict wordings pooled | 2026-11-11 14:21 |
| gpt-oss-20b, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | +4 | – | passed | Gemma Q8, three verdict wordings pooled | 2026-11-11 14:33 |
| gpt-oss-20b, production wording | bigger chat models | half A | 36 | +8 | product −2, test +7, infra +3 | dropped (class rule: product down by 2 (the rule allows 1 on A or B)) | production prompt, Gemma Q8 | 2026-11-11 14:57 |
| gpt-oss-20b, definitions | bigger chat models | half A | 36 | +7 | product 0, test +6, infra +1 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 14:57 |
| gpt-oss-20b, with hint | bigger chat models | half A | 36 | +4 | product 0, test +3, infra +1 | passed | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 14:57 |
| gpt-oss-20b, production wording | bigger chat models | half A | 36 | −2 | product −2, test +3, infra −3 | dropped | decision model with the environment question | 2026-11-11 14:57 |
| gpt-oss-20b, definitions | bigger chat models | half A | 36 | 0 | product −1, test +4, infra −3 | dropped | decision model with the environment question | 2026-11-11 14:57 |
| gpt-oss-20b, with hint | bigger chat models | half A | 36 | −1 | product 0, test +2, infra −3 | dropped | decision model with the environment question | 2026-11-11 14:57 |
| Gemma-26B-A4B, production wording | bigger chat models | half A | 36 | +12 | product +3, test +4, infra +5 | passed | production prompt, Gemma Q8 | 2026-11-11 15:36 |
| Gemma-26B-A4B, definitions | bigger chat models | half A | 36 | +5 | product +4, test −1, infra +2 | passed | fuller class definitions, Gemma Q8 | 2026-11-11 15:36 |
| Gemma-26B-A4B, with hint | bigger chat models | half A | 36 | +4 | product +2, test 0, infra +2 | passed | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 15:36 |
| Gemma-26B-A4B, production wording | bigger chat models | half A | 36 | +2 | product +3, test 0, infra −1 | dropped | decision model with the environment question | 2026-11-11 15:36 |
| Gemma-26B-A4B, definitions | bigger chat models | half A | 36 | −2 | product +3, test −3, infra −2 | dropped | decision model with the environment question | 2026-11-11 15:36 |
| Gemma-26B-A4B, with hint | bigger chat models | half A | 36 | −1 | product +2, test −1, infra −2 | dropped | decision model with the environment question | 2026-11-11 15:36 |
| gpt-oss-20b, definitions | bigger chat models | half B | 35 | +3 | product −3, test +4, infra +2 | dropped (class rule: product down by 3 (the rule allows 1 on A or B)) | fuller class definitions, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, definitions | bigger chat models | both halves (A+B) | 71 | +10 | product −3, test +10, infra +3 | dropped (class rule: product down by 3 (the rule allows 2 on A+B)) | fuller class definitions, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, with hint | bigger chat models | half B | 35 | 0 | product −2, test 0, infra +2 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, with hint | bigger chat models | both halves (A+B) | 71 | +4 | product −2, test +3, infra +3 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, production wording | bigger chat models | half B | 35 | +4 | product −3, test +6, infra +1 | dropped (class rule: product down by 3 (the rule allows 1 on A or B)) | production prompt, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, production wording | bigger chat models | both halves (A+B) | 71 | +12 | product −5, test +13, infra +4 | dropped (class rule: product down by 5 (the rule allows 2 on A+B)) | production prompt, Gemma Q8 | 2026-11-11 16:10 |
| gpt-oss-20b, definitions | bigger chat models | both halves (A+B) | 71 | −2 | product −5, test +7, infra −4 | dropped | decision model with the environment question | 2026-11-11 16:10 |
| gpt-oss-20b, with hint | bigger chat models | both halves (A+B) | 71 | −2 | product −2, test +3, infra −3 | dropped | decision model with the environment question | 2026-11-11 16:10 |
| Winnow-E4B, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | −2 | – | dropped | Gemma Q8, three verdict wordings pooled | 2026-11-11 16:22 |
| Wald-4B, three wordings | bigger chat models | 12 hard jobs (other), not drawn | – | −2 | – | dropped | Gemma Q8, three verdict wordings pooled | 2026-11-11 17:32 |
| Gemma-26B-A4B, production wording | bigger chat models | half B | 32 | +3 | product +1, test 0, infra +2 | dropped (coverage: 3 of 35 jobs timed out, so it was scored on 32 only and not counted as a pass) | production prompt, Gemma Q8 | 2026-11-11 20:27 |
| Gemma-26B-A4B, definitions | bigger chat models | half B | 32 | +2 | product +1, test −1, infra +2 | dropped | fuller class definitions, Gemma Q8 | 2026-11-11 20:27 |
| Gemma-26B-A4B, with hint | bigger chat models | half B | 32 | −4 | product −2, test −3, infra +1 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 20:27 |
| Gemma-26B-A4B, production wording | bigger chat models | both halves (A+B) | 68 | +15 | product +4, test +4, infra +7 | dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) | production prompt, Gemma Q8 | 2026-11-11 20:28 |
| Gemma-26B-A4B, definitions | bigger chat models | both halves (A+B) | 68 | +7 | product +5, test −2, infra +4 | dropped (coverage: 3 of 71 jobs timed out, so it was scored on 68 only and not counted as a pass) | fuller class definitions, Gemma Q8 | 2026-11-11 20:28 |
| Gemma-26B-A4B, with hint | bigger chat models | both halves (A+B) | 68 | 0 | product 0, test −3, infra +3 | dropped | decision model's estimate as a hint, Gemma Q8 | 2026-11-11 20:28 |
| Gemma-26B-A4B, production wording | bigger chat models | both halves (A+B) | 68 | −1 | product +3, test −2, infra −2 | dropped | decision model with the environment question | 2026-11-11 20:28 |
| Gemma-26B-A4B, production wording | bigger chat models | half B | 32 | −3 | product 0, test −2, infra −1 | dropped | decision model with the environment question | 2026-11-11 20:28 |
| Kev-9B, environment rule | other decision models | half A | 36 | −10 | product −8, test −5, infra +3 | dropped | decision model with the environment question | 2026-11-10 17:36 |
| Kev-9B, label question | other decision models | half A | 36 | +3 | product −1, test 0, infra +4 | passed | decision model, label question only | 2026-11-10 17:36 |
| Kev-4B, environment rule | other decision models | half A | 36 | −8 | product −6, test −5, infra +3 | dropped | decision model with the environment question | 2026-11-10 17:51 |
| Kev-4B, label question | other decision models | half A | 36 | −7 | product −8, test −5, infra +6 | dropped | decision model, label question only | 2026-11-10 17:51 |
| Clef-Flash, environment rule | other decision models | half A | 36 | −3 | product −4, test +4, infra −3 | dropped | decision model with the environment question | 2026-11-10 18:12 |
| Clef-Flash, label question | other decision models | half A | 36 | −2 | product −5, test +3, infra 0 | dropped | decision model, label question only | 2026-11-10 18:12 |
| d3-flash, environment rule | other decision models | half A | 36 | −3 | product −5, test +4, infra −2 | dropped | decision model with the environment question | 2026-11-10 19:00 |
| Wald-4B as a decision model | other decision models | half A, not drawn | 0 | 0 | product 0, test 0, infra 0 | dropped | decision model with the environment question | 2026-11-10 19:00 |
| d3-flash, label question | other decision models | half A | 36 | −2 | product −5, test +4, infra −1 | dropped | decision model, label question only | 2026-11-10 19:01 |
| Kev-9B, label question | other decision models | half B | 35 | −2 | product −4, test +1, infra +1 | dropped | decision model, label question only | 2026-11-10 19:47 |
| Kev-9B, environment rule | other decision models | half B | 35 | −7 | product −9, test −4, infra +6 | dropped | decision model with the environment question | 2026-11-10 19:47 |
| Kev-9B, label question | other decision models | half B | 35 | −5 | product −5, test +1, infra −1 | dropped | decision model with the environment question | 2026-11-10 19:47 |
| Kev-9B, label question | other decision models | both halves (A+B) | 71 | +1 | product −5, test +1, infra +5 | dropped | decision model, label question only | 2026-11-10 19:47 |
| Kev-27B, environment rule | other decision models | half A | 36 | +2 | product +1, test 0, infra +1 | dropped | decision model with the environment question | 2026-11-11 05:41 |
| Kev-27B, label question | other decision models | half A | 36 | +5 | product +1, test 0, infra +4 | passed | decision model, label question only | 2026-11-11 05:41 |
| Kev-27B, label question | other decision models | half B | 35 | +4 | product −2, test 0, infra +6 | dropped (class rule: product down by 2 (the rule allows 1 on A or B)) | decision model, label question only | 2026-11-11 07:00 |
| Kev-27B, environment rule | other decision models | half B | 35 | +1 | product −3, test 0, infra +4 | dropped | decision model with the environment question | 2026-11-11 07:00 |
| Kev-27B, label question | other decision models | both halves (A+B) | 71 | +9 | product −1, test 0, infra +10 | passed | decision model, label question only | 2026-11-11 07:00 |
| Kev-27B, environment rule | other decision models | both halves (A+B) | 71 | +3 | product −2, test 0, infra +5 | dropped | decision model with the environment question | 2026-11-11 07:00 |
| hint plus environment override | combination rules | half B | 33 | +2 | product +1, test +1, infra 0 | dropped | decision model with the environment question | 2026-11-10 11:37 |
| hint plus environment override | combination rules | half A | 33 | +1 | product +1, test 0, infra 0 | dropped | decision model with the environment question | 2026-11-10 11:37 |
| nearest labelled precedent rule | combination rules | half A | 36 | −1 | product 0, test −1, infra 0 | dropped | decision model with the environment question | 2026-11-10 11:41 |
| nearest labelled precedent rule | combination rules | half B | 35 | 0 | product −1, test +1, infra 0 | dropped | decision model with the environment question | 2026-11-10 11:41 |
| investigation clone results | combination rules | no screen run, not drawn | – | – | – | dropped (coverage) | – | 2026-11-10 11:44 |
| hint or definitions plus environment override | combination rules | both halves (A+B), not drawn | – | – | – | dropped | – | 2026-11-10 11:44 |
| yes/no overrides | combination rules | no screen run, not drawn | – | – | – | dropped | – | 2026-11-10 11:44 |
| six new yes/no questions | combination rules | half A, not drawn | – | – | – | dropped | – | 2026-11-10 14:01 |
| Kev-27B and elephant-1 averaged | combination rules | half A | 36 | −1 | product −1, test −1, infra +1 | dropped | decision model with the environment question | 2026-11-11 05:43 |
| Kev-27B and elephant-1 averaged | combination rules | half B | 35 | +3 | product −1, test 0, infra +4 | passed | decision model with the environment question | 2026-11-11 07:00 |
| Kev-27B and elephant-1 averaged | combination rules | both halves (A+B) | 71 | +2 | product −2, test −1, infra +5 | dropped | decision model with the environment question | 2026-11-11 07:00 |
| verdict asked again | sampling and voting | half A | 36 | 0 | product +1, test −1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 15:02 |
| verdict in a fresh conversation | sampling and voting | half A | 36 | +3 | product 0, test +3, infra 0 | passed | fuller class definitions, Gemma Q8 | 2026-11-10 15:02 |
| verdict in a fresh conversation | sampling and voting | half A | 36 | −4 | product −1, test +1, infra −4 | dropped | decision model with the environment question | 2026-11-10 15:02 |
| verdict in a fresh conversation | sampling and voting | both halves (A+B) | 71 | +3 | product −1, test +4, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 15:41 |
| verdict in a fresh conversation | sampling and voting | half B | 35 | 0 | product −1, test +1, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 15:41 |
| majority of five samples | sampling and voting | half A | 36 | +1 | product +1, test 0, infra 0 | dropped | fuller class definitions, Gemma Q8 | 2026-11-10 16:30 |
| flag: decision model disagrees with Gemma | flags | half A, not drawn | – | – | – | dropped | – | 2026-11-10 13:14 |
| flag: the five samples disagree | flags | half A, not drawn | – | – | – | dropped | – | 2026-11-10 16:31 |
| flag: top vote 3 of 5 or less | flags | half B, not drawn | – | – | – | dropped | – | 2026-11-10 17:20 |
| TF-IDF baseline | classic baseline | half A | 36 | +9 | product +2, test +4, infra +3 | passed (passed A, then a leakage check showed it had learnt to recognise tickets (jobs of one ticket on both sides of its split); trained only on earlier jobs it was level with the decision rule) | decision model with the environment question | 2026-11-10 11:34 |
| TF-IDF leakage check | classic baseline | both halves (A+B), not drawn | – | – | – | dropped | – | 2026-11-10 11:44 |
Bigger models did better than Gemma, but no better than the rule in the next section. On the evening of 11 November I stopped the slow models. I felt we were not investing our time most efficiently.
A second model
Gemma is a chat model: it writes text, and the script parses a class out of it. A decision model takes a question and a list of allowed answers, and returns a probability for each answer. I used elephant-1, a 9B decision model that runs on the same llama.cpp.
Data
- Chat model, Gemma 4 E4B:
- In: the prompt with the job state.
- Out: a block of thinking tokens, about 90% of what it generates, then text ending with “Classification: Product bug”.
- The class is parsed from the last lines.
- The probability of the answer token after all that thinking is about 1.0 whatever the answer, so it says nothing about confidence.
- Decision model, elephant-1 (9B):
- In: the same job state, a question and a fixed list of answers.
- Out: one probability per answer, for example product bug 0.43, test issue 0.35, infrastructure 0.22.
- A second yes/no question, “did the test environment fail?”, gets 0.06 against a threshold of 0.3.
- Rule used in the article: an environment answer of at least 0.3 means infrastructure, otherwise whichever of product bug and test issue has the higher probability. Here 0.06 is below 0.3, so the answer is product bug.
On its own it was no better than Gemma: on the gold jobs it was right on 42 of 95 against Gemma’s 50. But one question picked out infrastructure. Asked “did the test environment fail?” on the 116 jobs I tuned on, it gave “yes” a probability of 0.3 or more on 19 of the 30 infrastructure jobs and on 5 of the 86 others. I read that threshold off the data after seeing it.
Data
| Hindsight label | Jobs | At or above 0.3 | Below 0.3 | Total |
|---|---|---|---|---|
| product bug | 128 jobs the threshold was not read from | 2 | 68 | 70 |
| test issue | 128 jobs the threshold was not read from | 3 | 13 | 16 |
| infrastructure | 128 jobs the threshold was not read from | 22 | 20 | 42 |
| infrastructure | 116 jobs the threshold was read from, internal jobs included | 19 | 11 | 30 |
| product bug or test issue | 116 jobs the threshold was read from, internal jobs included | 5 | 81 | 86 |
| Probability | product bug | test issue | infrastructure |
|---|---|---|---|
| 0.0–0.1 | 28 | 13 | 6 |
| 0.1–0.2 | 35 | 0 | 14 |
| 0.2–0.3 | 5 | 0 | 0 |
| 0.3–0.4 | 1 | 2 | 1 |
| 0.4–0.5 | 1 | 1 | 3 |
| 0.5–0.6 | 0 | 0 | 5 |
| 0.6–0.7 | 0 | 0 | 6 |
| 0.7–0.8 | 0 | 0 | 2 |
| 0.8–0.9 | 0 | 0 | 5 |
| 0.9–1.0 | 0 | 0 | 0 |
That left a rule with two questions and no Gemma in it. If the environment answer is 0.3 or more, the class is infrastructure. Otherwise it is whichever of product and test elephant-1 rates higher. The package for its server is merged (openQA #8102); the change to the script is open (#814).
I also handed elephant-1’s answer to Gemma as a hint. That passed its screen on the small sample, but the steered Gemma repeated elephant-1’s label on 59 of 71 jobs, so the gain was elephant-1’s. What surprised me most was how one can steer an LLM to an incorrect conclusion.
The test and the audit
To test the rule I froze it first. I wrote the rule, the threshold and the scoring into a file, and then ran it on 128 o3 jobs of the silver set1. The comments posted in production had the class right on 66 of the 128. The rule had it right on 93.
Then I audited the labels. All 128 were silver labels. 108 of them had been made with hindsight, 36 of the 42 infrastructure labels among them. A rerun that passed is a fair sign of an infrastructure problem. It is not something a model can see in the log of the failed job. And the labeller had worked from the same written class definitions that I later gave to elephant-1.
Three checks do not depend on those labels. The plainest uses 35 gold jobs, labelled by people. The rule got 20 right. Production’s model and prompt, run again on the same input, got 19. These jobs were among the ones I had chosen the threshold on, which should have favoured the rule. The other two checks are in the lower half of the figure.
Data
| Set | Jobs | Posted comments | Rule |
|---|---|---|---|
| All 128 jobs (label set by a model) | 128 | 66 | 93 |
| All jobs: product bug | 70 | 38 | 58 |
| All jobs: test issue | 16 | 9 | 13 |
| All jobs: infrastructure | 42 | 19 | 22 |
| Label resting on a passing rerun | 23 | 16 | 13 |
| Label resting on a passing rerun: test issue | 2 | 2 of 2 | 2 of 2 |
| Label resting on a passing rerun: infrastructure | 21 | 14 of 21 | 11 of 21 |
| Other hindsight label | 85 | 40 | 68 |
| Other hindsight label: product bug | 60 | 32 of 60 | 50 of 60 |
| Other hindsight label: test issue | 10 | 5 of 10 | 10 of 10 |
| Other hindsight label: infrastructure | 15 | 3 of 15 | 8 of 15 |
| Label made without hindsight | 20 | 10 | 12 |
| Label made without hindsight: product bug | 10 | 6 of 10 | 8 of 10 |
| Label made without hindsight: test issue | 4 | 2 of 4 | 1 of 4 |
| Label made without hindsight: infrastructure | 6 | 2 of 6 | 3 of 6 |
| Gold jobs: label set by a person (not among the 128; among the jobs the threshold was chosen on) | 35 | 19 (replay of production) | 20 |
| Gold jobs: product bug | 19 | 13 of 19 (replay) | 14 of 19 |
| Gold jobs: test issue | 16 | 6 of 16 (replay) | 6 of 16 |
| Passed on a rerun (only partly among the 128): said infrastructure | 50 | 20 | 16 (both said it: 12) |
| Bug reference from a reviewer (among the 128; 2 bugs in all, one on 12 jobs; all product bugs) | 14 | 9 | 10 |
So the rule agrees much better than production with a hindsight labeller that shares its vocabulary, and no better with people.
The rerun
If an infrastructure label often means that a later run passed, then the best infrastructure detector on offer is the rerun. Of the 299 silver jobs, 188 were first failures: the previous run of the same scenario had not failed. 67 of those passed when they were run again in the same build2.
The hook starts investigation jobs only on BISECT: YES, and it retried 8 of the 67. The same comment also
said whether to restart the job: Restart: YES on 28 of the 67. No code acted on that line.
#821 starts one retry job when the comment says Restart: YES.
Data
| Jobs | Count |
|---|---|
| Commented jobs | 299 |
| First failures | 188 |
| Not a first failure | 111 |
| First failures: passed on a rerun | 67 |
| First failures: rerun, failed again | 44 |
| First failures: never rerun | 77 |
| Retry rule | Jobs it retries | Of the 67 that passed on a rerun, it covers |
|---|---|---|
| The hook as it was (its investigation retries) | 35 | 8 |
| Restart: YES | 62 | 28 |
| Restart: YES or BISECT: YES | 105 | 44 |
| Every first failure | 188 | 67 |
Pull requests
I opened 28 pull requests against os-autoinst-scripts and openQA between 25 October and 13 November. 22 are merged. The work is tracked in a public ticket.
| topic | merged | open |
|---|---|---|
| Prompt and verdict rules | #789 #790 #791 #795 #798 #801 #802 | |
| Comments and the hook | #788 #796 #797 openQA #8023 | #817 #819 openQA #8113 |
| Prompt length | #799 #800 | #815 |
| Routing and the model server | #808 openQA #8069 openQA #8070 openQA #8071 | |
| Groups nobody reviews | #810 | |
| Decision model | openQA #8102 | #814 |
Retry on Restart: YES | #821 | |
| CI | #792 #809 #816 |
What I would do differently
- Count the labels before the first experiment, per class and per ticket. On day one that would have shown the 30-point limit.
- Measure the noise floor next and size every pass rule from it.
- Ask of every label where it comes from.
- Read every field the system already produces before building anything to replace it.
How this was done
I wrote the scripts and ran the experiments with an AI coding assistant. A frontier model made the silver labels, checked the gold tickets that did not map cleanly, and was the larger model in the comparison on Gemma’s input.
The data comes from o3, which is public, but job pages there expire, so I link none. Parts of several sets came from an internal instance and appear as counts only. The numbers behind every figure are in the repository of this site.
As for the comments themselves, they seem useful. I hope the open pull requests get merged.
None of the 128 was among the jobs the threshold had been read from, though I had partly worded the question on the same pool, and 12 share a test and a build with jobs I tuned on. The file named two versions: as the main rule, one that asks the product-or-test question with the options in three orders, and this two-question rule as the cheaper fallback. Both met the bar written in the file, with 95 and 93, and I report the cheaper one. ↩︎
The comments on the pull requests say 68. One of those reruns had failed and was set to passed by a
force_resultlabel after a retry of it passed. ↩︎
**LLM Investigation summary:**
* boot_to_desktop: The test failed because no candidate needle matching the expected desktop environment or display manager was found.
The `gnuhealth` test failed in the `boot_to_desktop` module because the system failed to find a matching screen (`needle`) that contained tags such as `generic-desktop` or `displaymanager`. This indicates that the expected graphical environment did not load successfully during the boot process.
Classification: Product bug BISECT: NO — 18 other job(s) in this build fail in the same module Restart: NO — the needle does not match the screen
This comment has been generated by [openqa-llm-investigate](https://github.com/os-autoinst/os-autoinst-scripts/blob/master/openqa-llm-investigate). flag:verbatim