Comparison of 35B LLM Against Typed-Decision Models on Real RFQs

Overview

This study compares Jev, Qwen3.5-35B-A3B, and Laya 421M on 12,000 U.S. federal IT solicitations. The goal was not just to measure accuracy but to determine how much of the queue could be safely automated based on confidence. The primary-class accuracies were 91.9% for Jev, 89.6% for Qwen3.5-35B-A3B, and 78.0% for Laya 421M. However, the decisive factor was not raw accuracy-it was whether the models could reliably signal when their answers were safe enough to automate.

Model Execution Paths

All three models received inputs formatted as typed-decision bundles. The execution environments differed significantly:

Path	Model	Execution
Jev	TypeSafe System One, jev-1.13.0	Hosted API with calibrated probabilities
Qwen	Qwen3.5-35B-A3B-FP8	On-prem vLLM with structured JSON
Laya	Convai Laya, 421M parameters	Open weights on an RTX 2000 Ada laptop GPU

Jev and Laya received byte-identical typed-question bundles, while Qwen received a strict JSON schema whose answers were mapped onto a shared taxonomy. Qwen produced 69 permanently malformed responses, leaving 11,931 rows in the all-source paired set.

Ground Truth and Evaluation Design

Ground truth came from behavioral evidence-the actual quotations made by a reseller's sales team. Two data sources were collected:

• Single-class opportunities: When a sales rep quoted an opportunity, the product types on the latest quote lines provided the composition label, yielding 741 paired opportunities with a single unambiguous primary class.
• Fulfillment mode: Configurator fingerprints and distributor information on quote lines produced 634 paired labels.

The latter gold standard reflects real downstream behavior but is not perfect truth. Quoted opportunities represent a selected subset, with Hardware comprising 77% of the single-class rows, and fulfillment labels including an unvalidated heuristic.

Results Summary

Model	Primary Accuracy	Calibration Error	Auto-accepted @ 95% Precision Target	Fulfillment Accuracy
Jev	91.9%	0.049	86.5%	65.0%
Qwen3.5-35B-A3B	89.6%	n/a (three confidence levels not reached)	71.0%	71.0%
Laya 421M	78.0%	0.322	45.7%	45.7%

Jev beat the much larger LLM by 2.3 percentage points on the primary class. Laya's shipped defaults trailed by 13.9 points. The biggest difference was calibration: Jev's expected calibration error was 0.049, allowing it to accept 641 of 741 gold rows at a 0.94 confidence cutoff (86.5% coverage with 96.7% observed precision). The Wilson 95% lower bound remained at or above the 95% target along the cutoff envelope. Qwen reported three prompt-defined confidence levels, but its "high" bucket covered 97.8% of rows with only 90.1% precision-no bucket reached the 95% target.

Calibration Changed the Winner

A classifier that is slightly more accurate but poorly calibrated leaves you without a defensible automation boundary. At a confidence cutoff of 0.94, Jev accepted 641 of 741 gold rows (86.5% coverage, 96.7% observed precision), keeping the system within the 95% precision target. By contrast, Qwen's buckets were too broad-none reached the target. Laya produced 705 distinct confidence values on 741 rows, but the granularity wasn't sufficient to meet the target at any useful cutoff.

Elaborate question structures and additional attachment excerpts did not meaningfully improve the top-level task. All nine variants of Jev (including various presence questions, subclass choices, hierarchical selections, titles, descriptions, and line-item additions) landed between 91.2% and 91.9% on the same 741 rows. The key finding was that the question bundle consumed most of the tokens, and the rare attachment excerpt appeared on only 5.6% of rows.

Queueing Rules Matter More Than Accuracy

The original design sent every row with any raised flag to human review. Under this policy, Any flag forces review resulted in 26.6% of rows going to human review with 93.5% primary precision on scored accepted rows. When flags were recorded as attributes instead of automatic blockers, the remaining review queue was dominated by genuine Jev-Qwen disagreements.

One queueing rule changed automation from 26.6% to 91.9%. Removing the blanket flag-and-review approach reduced the volume needing human attention by nearly two-thirds without sacrificing precision.

Fulfillment Mode Remains Challenging

Fulfillment mode was the hardest task for all three models. Qwen led with 71.0% accuracy, followed by Jev at 65.0% and Laya at 45.7%. Even Qwen's best performance fell far short of the operational target. Configured-build precision ranged from 19% to 36%, and the mixed category was essentially unsolved-a likely reason is that the distinction between distributor SKUs and an OEM-configured build typically lives inside a bill of materials, configurator quote, or spreadsheet attachment rather than in the notice text. Additionally, the fulfillment gold itself relies on an unvalidated "eight or more lines from one OEM" heuristic requiring further human validation.

Laya's Specific Challenges

Laya's 78.0% result describes shipped defaults on this task, not necessarily the model's ceiling. Several factors contributed to its performance:

• Truncation issues: Laya's sequence packer reserved a 192-token question head. With seven options and definitions spread across hardware, software, support, and services categories, each definition was truncated to 25 tokens, shrinking the instruction to as few as eight tokens. This caused option definitions to be silently cut off-Laya called 105 of 572 Hardware rows "Software," whereas Jev made the error seven times and Qwen twenty times.
• Presence questions over-fired: Laya emitted an average of 4.05 components per row, compared with 1.95 for Jev and 1.16 for Qwen.
• RFI flags: Laya's RFI and text-insufficient flags fired on 63% of rows, while discovery analysis found true RFI rates around 2-3%.

Planned Follow-Up Experiments

Six follow-up experiments are proposed to address identified gaps:

1. Increase the head budget
2. Implement warnings on truncation
3. Try typed-decision and multilingual checkpoints
4. Fit per-question thresholds
5. Split the primary question from the rest
6. Record actual token counts

These are testable engineering questions and welcomes contributions.

Cost and Operational Behavior

For all 12,000 opportunities:

Source	Cost / Compute	Observed Operation
Jev	$0.78 at list input-token price p50; 185 ms (p50), 273 ms (p95); zero errors	Hosted operations
Qwen3.5-35B	~80 minutes on one shared GPU; 2.7 rows/s; 69 malformed responses	On-prem vLLM
Laya 421M	Single-row on RTX 2000 Ada laptop; p50 299 ms, p95 576 ms; zero errors	Unbatched single-row

These are different deployment products, so the comparison isn't a simple price race. Jev includes hosted operations, Qwen used an existing on-prem cluster, and Laya ran unbatched at one row at a time.

Agreement Between Models

Jev and Qwen agreed on 91.0% of the 11,931 paired rows. The remaining 1,068 disagreements clustered around specific categories: Hardware versus Other, Hardware versus Software, Software versus Maintenance & Support, and Services versus Software or Other. Rather than reviewing random rows, these disagreements formed a valuable blind-adjudication pool.

Limitations

Several important caveats should be noted:

• Quote gold measures agreement with what reps quoted, not objective truth.
• Hardware dominates the evaluation set; Services has only six rows and Maintenance & Support has 32.
• Fulfillment gold contains an unvalidated heuristic.
• Subclass, lifecycle, and domain answers lack human gold.
• Laya ran single-row with default configuration, no threshold tuning, and one checkpoint.
• Qwen's three-valued confidence was imposed by the prompt.
• The 69 malformed Qwen rows are excluded from paired metrics and are not random.
• The study covers one organization, one domain, one Jev version, and one measurement period-boundaries are explicit, even if the benchmark doesn't pretend to be universal.

Reproducibility

Everything publishable is open:

• Repository: bhushankinge/jev-laya-classification-bench
• Full report: classification experiment report
• Aggregate metrics: results/e2-full
• Experiment runbook: pipeline/RUN.md
• Companion CUDA study: laya-cuda-bench

The repository includes 52 passing tests, deterministic figure generation, a citation file, and the pre-registered selection rule. For production classifiers, the author is particularly interested in whether teams will optimize for raw accuracy or for the largest slice of traffic that can be automated at a defensible precision bound.