justadev

This page runs from 150 metres down to 225 metres down · 16.1 bar at the top

Twilight zone. Faint blue light, too weak for plants. Many animals here make their own light.

Research

What I measure about my AI agents, and what the numbers say. Each finding has a question, a method, a result and a caveat. Every number has its window.

Agent measurements taken .

Estimates against reality

Result, live
In the last 30 days: Claude Opus 5.5 took 0.47x its estimate over 626 tasks; Codex gpt-6-astra took 0.53x its estimate over 225 tasks; Claude Sonnet 5.5 took 0.38x its estimate over 17 tasks; Codex gpt-6-luna took 0.27x its estimate over 8 tasks; Claude claude-fable-5-2 took 0.34x its estimate over 4 tasks; Codex gpt-reserve took 0.40x its estimate over 3 tasks; Claude Opus 4.8 took 0.60x its estimate over 1 task. Every one of them finished faster than it estimated. In plain terms, for Claude Opus 5.5: an hour estimated took 28 minutes (median). Only 0% to 18.2% of tasks landed within 20% or 5 minutes of their first estimate.
Question
When an agent estimates a task, how long does the task really take?
Method
Every non-trivial task is logged with an estimate when it starts and closed when it ends. An audit matches both events to the tool calls in the agent's own session transcript, then corrects or excludes any time that does not match. The calibration factor is the median of real time divided by the first estimate, per model that made the estimate.
Caveat
Only tasks with a matched transcript count. The model is the one that made the estimate, so a lane task logged by the orchestrator carries the orchestrator's model. Sonnet's factor rests on 10 tasks. The ledger started on 8 October 2026.

Each bar is real time divided by the first estimate. A bar that stops left of the 1.00x line finished faster than estimated. Faded bars rest on fewer than 20 tasks.

Claude Opus 5.5626 tasks0.47xCodex gpt-6-astra225 tasks0.53xClaude Sonnet 5.517 tasks, small sample0.38xCodex gpt-6-luna8 tasks, small sample0.27xClaude claude-fable-5-24 tasks, small sample0.34xCodex gpt-reserve3 tasks, small sample0.40xClaude Opus 4.81 task, small sample0.60x
Data table
Calibration per estimating model, last 30 days
ModelFactor (real / estimate)TasksWithin 20% or 5 min
Claude Opus 5.50.47x62614.1%
Codex gpt-6-astra0.53x22518.2%
Claude Sonnet 5.50.38x175.9%
Codex gpt-6-luna0.27x812.5%
Claude claude-fable-5-20.34x40%
Codex gpt-reserve0.40x30%
Claude Opus 4.80.60x10%

First-pass rate

Result, live
In the last 30 days the orchestrator recorded 393 verdicts after verifying lane work. 276 passed on the first try (70.2%). 117 needed a fix round or were rejected.
Question
How often does a lane's work pass verification without a fix?
Method
After a lane reports, the orchestrator verifies the work by running it and records one verdict for that run: pass, fixed or failed. The first-pass rate is the share of verdicts that are pass. The verdict is the only field in the ledger that an agent writes; every other number comes from harness records.
Caveat
Verdicts started on 10 October 2026. Four verdicts cannot rank a model. The models did not get the same tasks, so the rates compare outcomes, not raw ability.

Each bar is the share of verdicts that passed on the first try, from 0 to 100%. Faded bars rest on fewer than 10 verdicts, too few to rank.

Codex gpt-6-astran = 142 verdicts69.7%Claude Opus 5.5n = 96 verdicts77.1%Claude Opus 5.5n = 64 verdicts82.8%Claude Sonnet 5.5n = 38 verdicts73.7%Grok 4.7 Buildn = 37 verdicts43.2%Claude Sonnet 5.5n = 11 verdicts45.5%Codex gpt-reserven = 3, too few to rank0%Codex gpt-6-lunan = 2, too few to rank50%
Data table
First-pass rate per model, last 30 days
ModelFirst passVerdictsLane runs
Codex gpt-6-astra69.7%142335
Claude Opus 5.577.1%96127
Claude Opus 5.582.8%6467
Claude Sonnet 5.573.7%3882
Grok 4.7 Build43.2%3766
Claude Sonnet 5.545.5%1120
Codex gpt-reserve0%33
Codex gpt-6-luna50%210

Who does the work

Result, live
In the last 30 days: 714 runs and 562.5 busy hours across 19 models. Codex gpt-6-astra ran 335 of them (46.9%).
Question
Which models carry the load, in runs and in busy hours?
Method
A run is one delegated lane run, recorded by the harness that starts it, native subagent runs included. Busy hours are the time a model spent working inside its sessions, read from the session transcripts. An unknown value stays empty; nothing is estimated.
Caveat
A run without a thread id gets a model only when its working directory and start time match exactly one session within 15 seconds; the rest stay model unknown. Busy hours add up parallel sessions, so they can exceed the hours on the clock.
Time window

In the last 30 days: 714 runs and 562.5 busy hours across 19 models.

Each measure is scaled to its own longest bar, so compare runs with runs and hours with hours.

Codex gpt-6-astra335 runs190.2 h busyClaude Opus 5.5127 runs312.8 h busyClaude Sonnet 5.582 runs11.1 h busyClaude Opus 5.567 runsno recordGrok 4.7 Build66 runs38.8 h busyClaude Sonnet 5.520 runsno recordCodex gpt-6-luna10 runs3 h busyCodex gpt-reserve3 runs1.2 h busyGrok 4.71 run2.6 h busyCodex gpt-61 run0 h busyCodex gpt-6-codex1 run0 h busyCodex gpt-6.1-sol1 run0 h busyClaude claude-fable-5-20 runs1.4 h busyClaude Opus 4.80 runs1.4 h busyClaude Haiku 5.50 runs0 h busy
Data table
Runs and busy hours per model, last 30 days
ModelRunsDoneBusy hoursSessionsMedian run
Codex gpt-6-astra33585.4%190.230618.2 min
Claude Opus 5.512782.7%312.870747 min
Claude Sonnet 5.58289%11.124831.7 min
Claude Opus 5.56798.5%no record038.5 min
Grok 4.7 Build6692.4%38.88329.9 min
Claude Sonnet 5.520100%no record010 min
Codex gpt-6-luna1090%31318.4 min
Codex gpt-reserve3100%1.2424.4 min
Grok 4.710%2.6742.5 min
Codex gpt-610%030.1 min
Codex gpt-6-codex10%020.1 min
Codex gpt-6.1-sol10%040.1 min
Claude claude-fable-5-20no record1.44no record
Claude Opus 4.80no record1.42no record
Claude Haiku 5.50no record039no record

What an agent's own timing is worth

Result
Of 183 tasks, 175 matched their transcript and 1 needed a corrected time. 6 were excluded: 2 were closed within two minutes on an estimate of 15 minutes or more, so they were logged after the fact, and 4 stayed open after their session went idle for six hours. 1 had no supporting record at all; it came from Grok. The same audit counted 63 failed logging calls.
Question
When an agent reports when it started and finished, can the report be trusted?
Method
On 10 October 2026 the first full audit matched every logged task against the session transcripts of Claude Code and Codex. Each start and each close had to match a real tool call, in time and in text.
Caveat
One audit, one operator, about two days of tasks. It measures whether the log is honest, not whether the estimates were good.

nerfIndex: do AI models get worse?

Result
First daily scores, 29 September 2026: Qwen3.8 27B on Groq 77.8%, Qwen3 8B local 69%, gpt-oss-120b 65.2%, gpt-oss-20b 63.0%, Qwen2.5 3B local 46.8%. On 8 October 2026 the catalog held 476 models; among 143 base models with enough public benchmark coverage, Xiaomi MiMo-V2.6-Pro had the highest public score under $1 per million tokens.
Question
Do hosted AI models lose quality over time without notice?
Method
nerfIndex runs a fixed daily suite: 48 items over 8 dimensions (reasoning, instruction following, structured output, code, long context, knowledge, tool use and refusal), drawn from public datasets such as GSM8K, MMLU-Pro, HumanEval+ and IFEval. Temperature 0, seed 42, once a day per route. It also records price moves, incidents from 27 official status pages, limit changes and new models. A model is marked degraded only after ten complete runs at least 20 hours apart: the first eight set the baseline, and the drop must be at least 5 points, with a 99% bootstrap interval below zero and McNemar p < 0.01, on the latest two runs.
Caveat
The daily suite runs on free routes only, under a $0 inference budget, so it does not cover paid frontier models, and Anthropic models are excluded. One day of scores says nothing about a trend; a verdict needs the full ten runs.

Last check from this server: nerfindex.com answered HTTP 200 at .

How many price changes are real?

Result
On a read-only snapshot of production on 8 October 2026, the 30-day price feed went from 1,086 events to 403, and the last 24 hours from 169 to 108.
Question
Model catalogs change prices all day. How much of that is noise?
Method
nerfIndex folds a price flap into one event. A field that returns to its old value within 48 hours, or switches three or more times in 24 hours, folds the whole model-day. A real numeric move must hold for six hours before it appears.
Caveat
The thresholds are a choice, not a standard. A real change that reverts within 48 hours is folded as a flap.