We built the benchmark we'd want any AI estimation vendor to pass. Then we failed it.

Large IT projects run about 45% over budget on average

That figure comes from McKinsey and Oxford looking at more than 5,400 of them, and it is the reason a wave of tooling now promises to read your spec and tell you what it will cost. We were about to build one of those tools. Before writing the product, we wrote the test.

The claim is falsifiable, which is the useful thing about it. The effort-estimation research community has been publishing datasets with real recorded effort for decades. So we asked the version of the question that can come back "no": On public data with real logged effort, can a statistical engine beat the standard baselines and the human expert, out of sample, without leakage? It cannot. Not with any model class we threw at it.

The benchmark is public, MIT, and reproducible from a pinned environment.

The bar, frozen before any result

We fixed what "passing" meant and committed it to git before running each gate experiment. The git history is the pre-registration.

PRED(25) and MdAPE are the standard metrics in this literature, so the results sit next to published papers without translation. And 55% is deliberately lenient: it is a lower bar than the KPIs we would set on one organization's own history, because public datasets mix organizations and are messier by construction.

Nine datasets, two channels

Tabular project attributes (PROMISE Desharnais, COCOMO81, China, Kitchenham, Maxwell, Albrecht, plus SEERA) and the natural-language text of requirements (JOSSE, SiP, with Deep-SE as secondary evidence).

Four choices that make a negative worth reading

A disappointing number is not a finding. These are the parts that turn one into the other.

Time-ordered evaluation, never random

Random splits let the model peek at the future and inflate every metric. We used rolling-origin cross-validation on the dated datasets and leave-projects-out grouped k-fold on the task datasets. The second one is deliberately the cold-start case: a new project retrieves only from other projects' history, which is exactly the situation a vendor walks into with a new client.

An adversarial leakage audit

Plenty of dataset attributes secretly encode the outcome. SEERA's Requirements stability is computed from modifications during testing and deployment. Team continuity counts developers who left the project. Programmers' capability is built from realized work accuracy. None of that is knowable before a project starts. An independent three-way automated audit classified all 76 SEERA attributes against their published formulas, and we kept only the ones that survived unanimously: 51 of 76.

Conformalized intervals

We do not promise a point, we promise a range that holds. CQR wraps any model and calibrates a distribution-free interval, and we check the empirical coverage rather than the assumed one.

A stop rule committed in advance

Before seeing results we fixed this: if the honest feasibility ceiling, meaning the best out-of-sample PRED(25) across a flexible zoo (Gaussian process, gradient boosting, random forest, k-NN, TF-IDF), sits below 55% on more than one gate dataset, then 55% is unreachable on this data and the channel closes as "threshold unreachable, demonstrated". That last one is what converts "we couldn't get a good number" into "the number is not there".

Channel A: tabular

The best engine was a log-space Gaussian process. Gradient boosting, a size mean function and hierarchical pooling across datasets were all implemented and none of them helped, because the GP already captures size through its features.

Aggregated over the four gate development datasets (n=207) the engine reaches 43.5% [36.7, 50.2] and beats the log-size baseline by 12.1 points [5.3, 18.8]. That lift is real and confident. It is also not enough: no dataset's central estimate reaches 55%, and the ceiling's point estimate is below 55% on all five including the sealed holdout, which fired the stop rule.

I want to be careful about what that does and does not say. The tabular test blocks are small, 12 to 73 projects, and Maxwell's ceiling has an upper bound of 67.7%. We are not claiming 55% is impossible on any single small dataset. We are claiming its central estimate is below 55% everywhere we looked, on data whose noise floor is high: the best model's out-of-sample residual scatter is σ ≈ 0.6 in log space, which is roughly ±60% multiplicative, on every dataset. Richer models do not move that. There is nothing left to extract.

Channel B: the requirement text

The hypothesis was reasonable. Semantically similar requirements should take similar effort, so embed the text, retrieve nearest neighbours, predict a kernel-weighted mean.

No text method beats the text-blind median on real logged effort. On SiP it is a statistical tie. On JOSSE the entire ceiling, 15.0%, sits below the floor of 17.2%, which means the requirement text is worse than guessing the median. Cross-encoder reranking made it worse still, though that one is on us: the standard ms-marco reranker is a query-document model and task pairs are symmetric.

There is one positive in the whole channel. On Deep-SE story points the semantic channel beats the floor by 2.7 points [2.0, 3.4]. Text carries some signal. But story points are a team's own relative scale, not hours, and the signal does not transfer to logged time.

The likely mechanism is not mysterious. Logged time is dominated by who did the work, how often they were interrupted, and local logging habits. None of that is in the requirement text, and across organizations it is most of the variance. Two identically worded tasks take very different logged time on two different teams.

Two things survived

The intervals are honest

A nominal-90% conformal interval covers the truth 94.6% of the time [91.5, 97.7] on unseen projects, distribution-free. It over-covers at lower nominal levels, so the guarantee is kept but loose. The failure here is accuracy, not calibration, and those are different products. "Here is a range that holds 90% of the time" is deliverable today. "Here is the number" is not.

The human expert wins

Where a dataset records a strong expert estimate, the expert beats every model: Kitchenham 61.6%, JOSSE 45%, SiP 41.6%. Experts use team and codebase context that public features and public text simply do not contain. The one place the model holds its own is the SEERA holdout, where the recorded expert is itself weak at 25.4% and the model ties it.

The follow-up that surprised me

If the expert beats the model, the obvious product move is to blend them. Average the two, keep the best of both. We tested it properly and it does not work. Pooled over 10,551 paired projects, an inverse-variance blend loses to the expert alone with McNemar p < 0.0001, at power ≈ 1.00. That is a real negative, not a wide confidence interval. The diagnostic is that the model and expert residuals correlate, so the diversification a blend is banking on is largely fictitious.

Worth stating the scope: on SiP and JOSSE the "model" in that test is the Track B engine that had already failed to beat the text-blind floor, so this measures blending that model, not blending in general. The weighting machinery is model-agnostic. If some future model measures a smaller residual spread than the expert on a client's own calibration data, the same formula shifts weight toward it automatically.

The first version of this test, at G0, was under-powered: n=73, about 11 discordant pairs, power ≈ 0.16, McNemar p=0.2266. It showed a +6.8 point difference with a confidence interval of [-1.4, +15.1] that included zero. An earlier draft of our own report framed that as a positive direction. It is retracted in the repo, because absence of proof is not proof of presence either, and a 0.16-power test is not evidence of anything.

What this does not prove

Every gate split was a cross-organization cold start on public data. That is the hardest case, and it is precisely not where an estimation tool gets deployed. A single organization's own history, same team, same logging conventions, same codebase, is the regime public datasets cannot proxy. This benchmark does not measure it and I make no claim about it here. It is the obvious next experiment, and it needs data we do not have. If you have that history and it interests you, the README says how to reach us. What I can promise is the same discipline you can read in the repo, applied to your data, and a straight answer either way.

Run it yourself

git clone https://github.com/NaCode-Studios/metis-benchmark
cd metis-benchmark
python -m venv .venv && source .venv/bin/activate
pip install -e ".[models,semantic,plots,dev]" -c constraints-g0.txt
make reproduce-g0

constraints-g0.txt is a pip freeze of the environment the numbers were produced in, on CPython 3.14.4. Bootstrap seeds are fixed, but a library upgrade can still move third-decimal behaviour in the GP optimizer or the XGBoost splits, so exact reproduction needs the constraints file. Every number in this post is produced by those scripts.

The protocol, its amendments and the pre-registration timestamps are in reports/protocol.md , the per-phase results are in reports/results/ , and the full write-up with the errata section is reports/REPORT.md.

Why publish this

We wrote a test designed to be hard, ran it on ourselves, and it came back no. Publishing that is cheaper than the alternative, which is shipping a claim about cold-start estimation from a spec and finding out in front of a client. There is also a shortage of this. Negative results in applied ML mostly do not get written up, so the same hypothesis keeps getting re-tested privately by people who cannot see each other's results.

If you are evaluating an effort-estimation vendor, the benchmark is a thing you can point at and ask them to run.

Repo: github.com/NaCode-Studios/metis-benchmark. Issues are open, and disagreement with the method is the most useful kind.