Laya: replace your LLM-as-a-judge with a 322M-parameter decision engine

Originally published at AI Frontier Post. You have a support inbox, a moderation queue, or an agent that needs to route its own work. The decision you need is embarrassingly simple - which department, how angry, should this escalate - but the standard implementation is absurd: serialize the text into a prompt, send it to a frontier model, wait for tokens to dribble out, then regex the answer out of the prose and pray the JSON parses. It is slow, it costs money per decision, and the same input can produce different outputs on different days. Laya (NandhaKishorM/laya , Apache-2.0) is the opposite bet: a non-autoregressive decision engine. You hand it text plus a set of typed questions - choice (pick a label), score (rate an intensity), noul (yes/no as a probability) - and one forward pass through an encoder answers all of them at once. No tokens are generated; the response even reports output_tokens: 0 . The repo calls it "System 1" thinking, and the metaphor is apt: fast, typed, deterministic judgments instead of deliberative prose. Why this is blowing up now The numbers are startling. Laya was created on September 18, 2026, and when I checked the GitHub API on September 27 it sat at 26,639 stars and 2,326 forks - roughly 3,000 stars a day for a nine-day-old project, with commits landing the same day I tested it. That is not normal. It is the kind of spike you see when a project names a pain everyone already feels. And the pain is real: the "LLM-as-a-judge" pattern has quietly become one of the most expensive lines in production AI systems. Every classification, every guardrail check, every routing decision burns frontier-model tokens and adds a second or more of latency. Laya's pitch is that 95% of those decisions never needed a reasoning engine at all - they needed a fast classifier with typed outputs, confidence scores, and the honesty to abstain. The repo ships three checkpoints on Hugging Face (convaiinnovations/laya ): a 421M-parameter English model built on ModernBERT-large, a 322M-parameter multilingual model covering 100+ languages, and a typed-decisions variant. The project reports 33ms per question on a T4 GPU, dropping to 7.2ms per question when batched - numbers I will sanity-check on CPU below. What you'll need Python 3.10 or newer and pip (I used 3.12 on Linux; macOS and Windows work). PyTorch - the CPU build is fine. I ran the entire tutorial on a plain CPU-only VM; no GPU needed. About 1.5 GB of free disk for the checkpoint (the multilingual weights are ~650 MB) and ~1 GB of RAM at inference time. No API keys, no accounts, no GPU, no cost. Everything runs locally. The checkpoints download from Hugging Face on first use and are cached afterwards. About 20 minutes, most of it the one-time checkpoint download. Step 1: Install Laya pip install torch --index-url https://download.pytorch.org/whl/cpu pip install "laya==0.3.21" python -c "import laya; print(laya.version)" # 0.3.21 Install PyTorch first from the CPU index so you do not pull the multi-gigabyte CUDA build by accident - that is the one genuine footgun in the setup. Then pin laya==0.3.21 , the version I verified everything below against. The import prints 0.3.21 and you are ready; nothing has been downloaded yet except the package itself. Step 2: Your first decision - the CLI Laya ships a CLI that is the fastest way to feel the core idea. Run it with no flags first - this performs routing only, no checkpoint download: laya "The CEO is furious and the board is about to vote on a merger" You will see something like this (exact output from my run): Model : english Reason : English Latin text Detected : {"script": "latin", "script_profile": {"latin": 1.0}, "language": "en", "is_english": true, "language_undecided": false, ...} The router inspects the script and language of your text and picks a checkpoint - English text goes to the 421M English model, anything else to the 322M multilingual one. This routing decision costs nothing and needs no weights. Now ask a real question with the built-in triage preset (this downloads the checkpoint on first use, ~650 MB, one time): laya "My payment failed twice" --preset triage --model multilingual --device cpu Model : multilingual Reason : explicit model='multilingual' intent : technical_help (p=0.811) is_urgent : 0.002 frustration : 2.28 refund_requested: 0.004 churn_risk : 0.001 Read that output carefully, because it is the whole product in miniature. Five typed decisions - one choice (intent), one score (frustration on a 0-3 scale), three noul yes/no probabilities - answered in one forward pass, with a probability attached to every answer. The triage preset's five questions are defined once and reused; the available presets are email , guard , moderation , router , and triage . Nothing was generated token-by-token. Nothing cost you a cent. Laya's core idea: one encoder, one forward pass, all question heads answered in parallel. No autoregressive decoding. Step 3: Batch triage - including an honest miss Single tickets are a demo; the real use case is a queue. Put one ticket per line in a file and score the batch: printf 'My payment failed twice, I want my money back\nBonjour, je ne parviens pas a me connecter a mon compte\nThis is just a feature request, no rush\n' > tickets.txt laya --batch tickets.txt --preset triage --model multilingual --device cpu My exact results on CPU: # My payment failed twice, I want my money back intent : refund (p=1.000) is_urgent : 0.004 frustration : 2.16 refund_requested: 0.995 churn_risk : 0.006 # Bonjour, je ne parviens pas a me connecter a mon compte intent : technical_help (p=0.992) is_urgent : 0.000 frustration : 2.02 refund_requested: 0.000 churn_risk : 0.002 # This is just a feature request, no rush intent : cancellation (p=0.703) is_urgent : 0.009 frustration : 2.30 refund_requested: 0.005 churn_risk : 0.026 Two observations worth your attention. First, the French ticket - routed to the multilingual checkpoint - is classified correctly with 0.992 confidence. No translation step, no English-only fallback; that is the 100+ language claim working as advertised on at least one of them. Second, the third ticket is a miss: "This is just a feature request, no rush" gets labeled cancellation at p=0.703. I am showing you this deliberately, because it is the most important thing to understand about decision models: they are wrong sometimes, and the only honest design is one that tells you when it might be wrong. Notice the confidence - 0.703, far below the 0.99+ of the correct answers. That gap is load-bearing. It is what Step 4 is built on. On timing: the batch of 8 tickets I ran later took 27.5 seconds warm on CPU - about 3.4 seconds per ticket for five questions, or roughly 0.7 seconds per question. The project's reported 33ms per question assumes a T4 GPU; on CPU you should budget about a second per question per ticket after the one-time model load (~85 seconds on my VM). For a support queue or a nightly batch job, that is plenty fast. For per-keystroke latency, use a GPU. Step 4: The Python API - and teaching it to abstain The CLI is for exploration; production code uses the Router . Here is the complete triage loop, which I ran verbatim: from laya import Router, triage_questions router = Router(device="cpu") # checkpoints download on first use questions = triage_questions() # the 5-question preset from Step 2 tickets = [ "My payment failed twice, I want my money back", "Bonjour, je ne parviens pas a me connecter a mon compte", "URGENT: production database is down, customers cannot check out", ] out = router.predict(tickets, questions, model="multilingual", min_confidence=0.90) The result is a plain dict with four top-level keys - model , answers , usage , and routing . The routing block tells you exactly which checkpoint answered and why ("repo": "convaiinnovations/laya/multilingual", "reason": "explicit model='multilingual'" ), and usage reports input_tokens with output_tokens: 0 - the signature of a model that never generates. Each answer carries its type, its value, and its confidence. A choice answer includes the full probability distribution; a score includes the legend mapping numbers to meanings plus per-level probabilities; a noul includes the raw probability. Here is the frustration score for the payment ticket, exactly as returned: { "score": 2.3211, "legend": {"0": "calm and neutral", "1": "concerned but civil", "2": "clearly annoyed", "3": "very angry or using strong language"}, "probabilities": {"0": 0.0354, "1": 0.129, "2": 0.3148, "3": 0.5208}, "confidence": 0.2166, "answer_confidence": 0.5208, "low_confidence": true } That trailing "low_confidence": true is the min_confidence=0.90 parameter doing its job. The model's answer confidence (0.52) fell below my threshold, so Laya flagged the field instead of letting a shaky 2.32 flow silently into my pipeline. This is the opt-in abstention mechanism, and it is the difference between a demo and a system you can deploy: low-confidence fields come back flagged (and as None through the schema-driven decide() API), so your code can route them to a human or a bigger model. Note the honest subtlety - the same ticket's intent (refund, p=1.000) and refund_requested (0.9638) passed the threshold, so only the genuinely uncertain field was flagged. Abstention is per-question, not per-ticket. The Router: a lightweight language/script detector picks the right checkpoint per request - English text to the 421M model, everything else to the 322M multilingual one. Step 5: End to end - a confidence-gated escalation pipeline Now assemble the pieces into something you could actually ship: a triage function that auto-handles the easy tickets, escalates the urgent ones, and sends anything uncertain to a human review queue. def triage(ticket: str) -> dict: out = router.predict(ticket, questions, model="multilingual", min_confidence=0.90) a = out["answers"] # Abstain: any flagged field -> human review if any(v.get("low_confiden