What it costs, what it saves, and where it lost.
An A/B of the same agent on the same repository, with and without PyMolt attached, plus a control arm that never touches it. Every figure below is measured. The ones we cannot attribute are marked as such.
Where the tool wins, it wins on every axis at once.
The task: given a dependency upgrade, name every place in the codebase it can reach. It is not a greppable question, which is exactly why it separates the arms.
“Which call sites does this dependency change reach?”
same question, same model, same repoIt reported 35–38 contact points against a true 227 call sites across 55 distinct symbols, because grepping imports does not find call sites. The failure is not that it was slow. The failure is that it was confident and short by a factor of six.
The cheap answer is the right one.
Repriced from the measured runs to Anthropic list rates, 2026-08-27, on Opus 5. Across 252 measured turns.
On the contact-surface task the tool-less agent costs 9.3× more — for the wrong answer. On recon it costs very slightly less, because the question was easy enough that the tool did not earn its overhead.
How long a run actually takes
p50 of 5 repeats after a warmupThe same command, run against a real public repository.
$ pymolt scan .Two of the four failures above, in one unmodified open-source project: the container says 3.6 while the test matrix claims up to 3.11, and two dependencies never came from PyPI, so no public catalog can ever have rules for them. Clone it and run the same command — the output is checked into the repository.
flasgger's cold/warm `assess` shows no difference (571 vs 573 ms) because clearing .pymolt_cache does not clear uv's global cache. pyfolio does show it (1112 → 571 ms). A true cold number needs a clean ~/.cache/uv, which this run did not do.
blaze's `assess` fails outright, exit 3 — uv cannot resolve its decade-old pins. That is the tool correctly refusing, not a crash, but it is a failure and it is in the table.
Tokens are not the line item.
Even the pessimistic end of a 200,000-line migration is a fraction of the engineer time it displaces, and it arrives in days rather than months. No plausible choice of parameters makes the tokens the expensive part.
What a whole migration would cost
agent priced at Opus 5 list rates- 28.5 contact points per 1,000 LOC, from PyMolt's own scan of flasgger
- 40% of packages change version on a 3.9 → 3.12 move — 31 of 77
- $/turn across 252 measured turns
- Bun's $0.309 per line replaced
- 4 turns per affected call site
- 4 verification cycles of roughly 40 turns
- ~20 lines read or written per affected call site
- 3–8 engineer-hours per 1,000 LOC
- that flasgger's contact density generalises to other codebases
The shift is not cheaper migrations — it is migrations that were never worth doing. At $11k–60k a legacy upgrade competes with feature work and loses, which is why dead Python projects stay dead. At $100–2,000 the arithmetic changes, and the binding constraint moves from budget to confidence: can you prove the thing still works?
The cost model reproduces a published number to 0.3%.
Bun rewrote 1,448 Zig files, 535k lines, rewritten to Rust with Claude in 11 days, 6,502 commits, and published their token spend. Repricing those raw counts at list rates gives $165,500 against their reported ≈$165,000 — which validates the model, and incidentally reveals which model they ran.
That last figure is the one to keep. Prompt caching was not a 10% optimisation on that job — it was the difference between ≈$165,000 and $813,500, a saving of $648,000. Any cost model that ignores the cache ratio is wrong by roughly five times.
The oracle is the whole game
Them: Their test suite was TypeScript, so it survived the reimplementation untouched — a behavioral oracle that existed before the migration and cost nothing to keep.
Us: The same claim our payloads now make explicitly, because silence reads as completeness.
Agents take the shortcut unless something is watching
Them: Early runs stubbed functions instead of fixing them, with paragraph-long justifying comments. Reviewers were denied the implementer's reasoning on purpose.
Us: Given a list clipped at 20 items and a pointer to the full file, our agent answered from the clipped list in 2 of 3 runs. Same reflex, fixed structurally rather than by asking nicely.
One token number means nothing at any scale
Them: 72B cached reads against 5.9B uncached — a 12:1 ratio.
Us: We abandoned total_tokens for exactly this: cache reads dominated it, and the spread inside one arm (95k–157k) exceeded the gap between arms.
Don't let the agent decide whether to use the tool
Them: cargo check was wired in as the work-queue generator — a pipeline stage, not an option the model could skip.
Us: With the server merely attached, the agent made zero PyMolt calls on 2 of 3 tasks.
And the one place the two situations genuinely differ.
Bun got its oracle for free. Their suite was written in TypeScript — 1.39M expect() calls across 60,624 tests, zero skipped or deleted — so it survived the reimplementation untouched, and the whole rewrite was tractable because that many assertions already pinned the behavior. Take the oracle away and no amount of parallel agents, adversarial review or fuzzing tells you whether behavior was preserved. There is nothing to compare against.
The projects PyMolt is built for are exactly the no-oracle case. That is why contract report returns a BLIND number instead of pass/fail: it answers the question Bun never had to ask — how much of my behavior do my tests actually pin?
Five results that argue against us.
A benchmark with no losses in it is a brochure. These are in the same run as everything above.
A regression we caused ourselves
attributable to our changeRewriting the tool descriptions to lead with the behavioral contract pulled the agent's attention off the cheap recon step. The plugin arm fell from 1.00 to 0.84 on T1 and stopped calling scan entirely — 1 call to 0 — while the control held at 1.00.
A win we cannot claim
control moved identicallyT2's score went from 0.24 to 0.98 in the PyMolt arms — and by exactly the same amount in the control, which never touches PyMolt. That makes it drift between runs, not a result. Removing the 20-item truncation was still the right fix, and the full 31-row list now costs fewer tokens than the clipped JSON did (453 vs 643), but the score cannot be credited to it.
Attaching the server is not free
measured cost, always paidJust having the tools available costs +848 tokens on every request, up from +622 before the docstring rewrite. Paid on every turn of every session, whether or not a single PyMolt tool is called.
One fixture the tool cannot process
honest refusal, still a gapassess on blaze exits 3: uv cannot resolve its decade-old pins for a 3.9 target. The tool refuses rather than guessing, which is the intended behavior — but the practical result is that on this project that phase gives you nothing.
Agents do not reach for the tool on their own
fixed for one task, not in generalWith the server attached and a neutral prompt, the agent made zero PyMolt calls on T1 and T2 across both models — it shelled out instead. Rewriting the descriptions to say what an agent cannot do for itself moved T2 from zero calls to five, but the general lesson stands: a tool nobody is told to use does not get used.
What this is not
- n=3, on one 8,000-line fixture, one task family. This is indicative, not statistics.
- The control arm drifted substantially between rounds with no code touching it — enough to invalidate one of our own attributions. Only differences that separate from the control are claimed.
- Bun's figures are one real migration, self-reported, with no control group. You cannot tell from the article what the model contributed versus what a year of conventional work would have.
- None of the cost projections has been validated against a completed migration. They are a way to reason about order of magnitude, not a quote.