A 25-verifier panel measured an effective size of 1.00

A 25-verifier panel measured an effective size of 1.00

IDKMesh is an open-source research project (Apache-2.0, Python 3.11+) that asks how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into verified useful work. It is a research preview, not production software, and the repository is deliberately conservative about what it claims - every number below traces to a committed experiment record you can re-run.

The Core Finding

The assumption underlying many review systems is that each additional reviewer adds independent evidence. This assumption is measurable, and it did not hold. In practice, adding more verifiers does not proportionally increase the reliability of the collective judgment.

How It Works

The coordination model treats generation as cheap and verification as the scarce resource:

• Work arrives as a bounded Work Unit that must declare its security bounds.
• Replaceable workers attempt it. Multiple attempts are expected; candidates are disposable.
• A separate, verifier-owned evaluation plan judges those candidates.
• Worker completion is not acceptance. Results become reproducible evidence - result manifests, verification results, provenance digests.
• Integration is an explicit decision gated on that evidence, not on a green impression.

The bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and deliberately rejects four invalid ones - including a task with no security contract, and a worker result that accepts itself.

The Demonstration

Here is the bundled example report:

{
  "schema": "gate-audit-report-v0.1",
  "gate_id": "example-five-verifier-gate",
  "evidence_class": "synthetic",
  "inputs": {
    "candidates": 15,
    "verifiers": 5,
    "known_good": 8,
    "known_bad": 4
  },
  "panel": {
    "nominal_votes": 5,
    "mean_verifier_accuracy": 0.8,
    "mean_pairwise_error_correlation": 0.0916,
    "effective_votes": 1.6944,
    "heuristic_n_eff": 3.6588
  },
  "probes": {
    "total": 3,
    "breached": 2,
    "breach_rate": 0.667
  },
  "warnings": [
    "2/3 seeded known-bad probes were accepted by the panel"
  ]
}

Read the third key before the interesting one: "evidence_class": "synthetic". Those five reviewers are invented, and 1.69 is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence.

Observed vs Synthetic Measurements

Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain (tiny, small, large, extreme, duplicate) and accepts a candidate only if it matches a reference implementation on all of them. With 5 regions × 5 seeds = 25 verifiers running over a 72-candidate corpus whose ground truth comes from executing hidden tests, every verifier's Youden J is significantly positive after Bonferroni correction - mean accuracy 0.7956. So the panel genuinely works, individually.

Error correlation between them:

• Same region (declared dependent): mean ρ = +0.8924
• Different region (declared independent): mean ρ = +0.5263
• All pairs: mean ρ = +0.5873

Verifiers that share no declared attribute still share 53% of their errors. A metadata group boundary is not an independence boundary.

Then the part that matters:

The panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66×.

Why the Usual Correction Doesn't Rescue This

The familiar fix is to discount for correlation using N_eff = N/(1+(N-1)ρ).

Experiment E015 tested that heuristic across a parameter grid and found it exact at ρ=0 and ρ=1, wrong in between, and importantly - the sign of its error is not fixed. It is conservative where verifiers are weak (where conservatism is cheap) and optimistic where verifiers are accurate and share modest dependence (the regime you care about).

At p=0.90, ρ=0.125 the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error 0.000891, the real panel delivers 0.0125. That is fourteen times worse.

Scope Limitations

Being precise about scope, because it is easy to over-read:

• No AI review panel has been measured in this repository.
• E017's verifiers are programs, not LLMs. Whether LLM reviewers exhibit the same correlation structure is an open question - and the one I most want data on.
• It is a research preview. Not production software.
• It is not on PyPI yet, so installation is from a clone.

git clone https://github.com/MSKazemi/idkmesh && cd idkmesh
python -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements-phase0.txt
python scripts/demo.py

No model account or API key is needed. Repository: https://github.com/MSKazemi/idkmesh

If you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing gate-audit at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.