Inside NVIDIA Nemotron 3.5 Lightning: Building a (Local) Mixture-of-Experts Explorer
DEV Community

Inside NVIDIA Nemotron 3.5 Lightning: Building a (Local) Mixture-of-Experts Explorer

Introduction

Modern AI systems face a fundamental tension: users demand the deep reasoning and multi-step planning of large frontier models, yet running dense 30B+ parameter architectures for high-frequency subtasks (like code routing, tool parsing, and syntax validation) imposes massive computational and financial overhead.

To solve this, NVIDIA introduced Nemotron 3.5 Lightning, an open-weight 30-billion parameter Mixture-of-Experts (MoE) model that activates only 3 billion parameters per token.

This article explores nvidia-nemotron-lightning-35-test, a production-grade local explorer and testing framework designed to interact with, monitor, and visualize Nemotron 3.5 Lightning locally via Ollama.

Rather than just wrapping API endpoints, this project provides a clear architectural blueprint for building transparent, low-latency AI applications.

Implementation

The nvidia-nemotron-lightning-35-test project provides a robust framework and sandbox to explore, serve, and evaluate NVIDIA Nemotron 3.5 Lightning.

The repository architecture is structured around core demonstration modules, documentation sets, and utility scripts designed to test MoE behavior, speculative decoding, and application-level routing.

At the heart of the system is a Mixture-of-Experts (MoE) transformer backbone. Unlike dense models where all parameters are activated for every token, Nemotron 3.5 Lightning routes tokens through sparse expert layers, ensuring high representational capacity (30B total) combined with rapid execution speed (3B active parameters).

Architectural Philosophy: The MoE Paradigm & Efficiency

Unlike dense models that utilize their entire parameter set for every token generated, Nemotron 3.5 Lightning routes tokens dynamically through sparse expert layers.

This decoupled design provides the representational capacity of a 30B model with the execution speed of a 3B model.

+-------------------------------------------------------------------+
| Client / User Input                                               |
+-------------------------------------------------------------------+
β”‚ β–Ό
+-------------------------------------------------------------------+
| NeMo Switchyard / Router Layer                                    |
+-------------------------------------------------------------------+
β”‚ β–Ό
+-------------------------------------------------------------------+
| NVIDIA Nemotron 3.5 Lightning (MoE)                               |
| - 30B Total Parameters / 3B Active Parameters                     |
| - Multi-Token Prediction (MTP) & Speculative Decoding Support     |
+-------------------------------------------------------------------+
β”‚ β”‚ β–Ό β–Ό
+-------------------------+ +-------------------------+
| Draft Models            | | Serving Frameworks        |
| (DSpark / MTP / DFlash) | | (vLLM / llama.cpp / LoRA)|
+-------------------------+ +-------------------------+

To maintain complete intellectual honesty, the project’s application architecture enforces a strict boundary between hard telemetry (data directly measured from backend inference) and illustrative modeling (educational visual simulations):

Data Component Origin Status
Connection State & Model Metadata Ollama /api/version, /api/tags, /api/ps blog-post-V2.md Measured blog-post-V2.md
Generation Speed & Token Counts Ollama final streaming chunk (eval_count, eval_duration) blog-post-V2.md Measured blog-post-V2.md
Time to First Token (TTFT) Client-side performance timer blog-post-V2.md Measured blog-post-V2.md
Expert Activation Heatmap moe_demo/router_sim.py (Deterministic BLAKE2b simulation) blog-post-V2.md+ 1 Illustrative blog-post-V2.md

⚠️ Design Note on Ollama: Ollama does not currently expose internal per-token router telemetry (e.g., revealing exactly which expert ID fired for a specific token). The application explicitly flags its expert heatmaps as simulated, preserving user trust and technical accuracy.

Core Project Structure

The codebase is organized around a clean separation of concerns, isolating the GUI layer from core business logic and state computation:

nvidia-nemotron-lightning-35-test/
β”œβ”€β”€ Docs/ # System documentation and architectural references
β”œβ”€β”€ docs_tools/ # Automated document parsers and build scripts
β”œβ”€β”€ moe_demo/ # Core application packages
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ categories.py # Task heuristic classifiers and expert affinities
β”‚ β”œβ”€β”€ config.py # Environment-based configuration loader
β”‚ β”œβ”€β”€ engine.py # UI-agnostic request builder and stream handler
β”‚ β”œβ”€β”€ metrics.py # Nanosecond-to-throughput converters
β”‚ β”œβ”€β”€ ollama_client.py # Robust REST wrapper for Ollama APIs
β”‚ β”œβ”€β”€ router_sim.py # Deterministic top-6-of-128 routing simulator
β”‚ └── ui_components.py # Streamlit visual cards and heatmaps
└── app.py # Central Streamlit GUI entrypoint

Implementation Deep Dive

Environment Configuration (moe_demo/config.py)

Configuration is managed via environment variables with fallback support for local .env files. Using a frozen dataclass ensures settings remain immutable throughout the application lifecycle:

"""Runtime configuration. Values come from the environment / `.env` (see `.env.example`)."""
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
def load_env_file(path: Path | None = None) -> None:
    """Load KEY=VALUE lines from `.env` into os.environ without overriding real env vars. A tiny parser is used on purpose so the app has no extra dependency and also works when started with `streamlit run app.py` instead of `scripts/start.sh`."""
    path = path or ROOT / ".env"
    if not path.is_file():
        return
    for raw in path.read_text(encoding="utf-8").splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        value = value.strip().strip('"').strip("'")
        os.environ.setdefault(key.strip(), value)
@dataclass(frozen=True)
class Settings:
    base_url: str
    model: str
    keep_alive: str
    request_timeout: float
def get_settings() -> Settings:
    """Build settings from the environment (defaults only for non-sensitive values)."""
    load_env_file()
    return Settings(
        base_url=os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434").rstrip("/"),
        model=os.environ.get("OLLAMA_MODEL", "nemotron-3.5-lightning:30b-a3b-mlx"),
        keep_alive=os.environ.get("OLLAMA_KEEP_ALIVE", "15m"),
        request_timeout=float(os.environ.get("OLLAMA_TIMEOUT", "600")),
    )

Note: The default OLLAMA_TIMEOUT is set to 600 seconds to gracefully handle cold-starting a massive ∼19 GB∼19 GB model artifact without throwing premature timeouts.

Resilient Ollama Client (moe_demo/ollama_client.py)

The client wraps Ollama's REST API (/api/version, /api/tags, /api/ps, /api/generate, and /api/chat), translating raw HTTP errors into a clean exception hierarchy:

class OllamaError(Exception): ...
class OllamaConnectionError(OllamaError): ...
class OllamaModelNotFound(OllamaError): ...
class OllamaTimeout(OllamaError): ...

During streaming chat completions, NDJSON response streams are safely parsed into typed StreamEvent objects, separating thinking/reasoning blocks (thinking) from final generation tokens (content) and server statistics (done):

for line in resp.iter_lines():
    if not line:
        continue
    data = json.loads(line)
    if "error" in data:
        raise OllamaError(str(data["error"]))
    msg = data.get("message") or {}
    if msg.get("thinking"):
        yield StreamEvent("thinking", msg["thinking"])
    if msg.get("content"):
        yield StreamEvent("content", msg["content"])
    if data.get("done"):
        yield StreamEvent("done", stats=data)

UI-Agnostic Generation Engine (moe_demo/engine.py)

The engine converts user parameters into validated option payloads and manages conversation window constraints:

"""UI-independent generation logic: parameters -> Ollama request -> live metrics. Keeping this separate from Streamlit makes it unit-testable against a stub Ollama server."""
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Iterator
from .metrics import RunMetrics, from_stats
from .ollama_client import OllamaClient
THINK_MODES = ("Auto (per category)", "Off", "On", "Medium")
def build_options(temperature: float, top_p: float, num_ctx: int, num_predict: int) -> dict:
    """Map UI values to Ollama `options`. These are re-read on every request, so changing a slider affects the very next message."""
    if not 0.0 <= temperature <= 2.0:
        raise ValueError("temperature must be within 0..2")
    if not 0.0 < top_p <= 1.0:
        raise ValueError("top_p must be within (0, 1]")
    if num_ctx < 512:
        raise ValueError("num_ctx must be at least 512")
    return {"temperature": temperature, "top_p": top_p, "num_ctx": int(num_ctx), "num_predict": int(num_predict)}
def resolve_think(mode: str, category_think: str) -> bool | str:
    """Translate the UI reasoning selector into the value of Ollama's `think` field. The model accepts False, True or "medium" (see `ollama show`)."""
    chosen = category_think if mode.startswith("Auto") else mode.lower()
    if chosen == "off":
        return False
    if chosen == "medium":
        return "medium"
    if chosen == "on":
        return True
    raise ValueError(f"unknown reasoning mode: {mode!r}")
def build_messages(system_prompt: str, history: list[dict], prompt: str, use_history: bool, max_turns: int = 6) -> list[dict]:
    """System prompt + (optionally) the last `max_turns` turns of the conversation + new prompt."""
    messages: list[dict] = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    if use_history:
        for m in history[-max_turns * 2:]:
            if m["role"] in ("user", "assistant") and m.get("content"):
                messages.append({"role": m["role"], "content": m["content"]})
    messages.append({"role": "user", "content": prompt})
    return messages
@dataclass
class LiveUpdate:
    """Snapshot yielded while streaming."""
    content: str
    thinking: str
    chunks: int # streamed chunks ~ tokens (live approximation)
    elapsed: float
    ttft: float
    live_tps: float # approximate until the final exact metrics arrive
    final: RunMetrics | None = None
def stream_answer(client: OllamaClient, model: str, messages: list[dict], options: dict, think: bool | str, keep_alive: str) -> Iterator[LiveUpdate]:
    """Stream a reply and yield LiveUpdate objects; the last one carries exact `final` metrics."""
    start = time.perf_counter()
    first: float | None = None
    content, thinking, chunks = "", "", 0
    for ev in client.chat_stream(model, messages, options=options, think=think, keep_alive=keep_alive):
        now = time.perf_counter()
        if ev.kind in ("content", "thinking"):
            if first is None:
                first = now - start
            chunks += 1
            if ev.kind == "content":
                content += ev.text
            else:
                thinking += ev.text
            gen_time = max(now - start - (first or 0), 1e-6)
            yield LiveUpdate(content, thinking, chunks, now - start, first or 0.0, chunks / gen_time if chunks > 1 else 0.0)
        elif ev.kind == "done":
            wall = now - start
            final = from_stats(ev.stats, ttft=first or wall, wall=wall)
            yield LiveUpdate(content, thinking, chunks, wall, first or wall, final.tokens_per_second, final)
def reasoning_share(thinking: str, content: str) -> float:
    """Fraction of streamed characters that belong to the reasoning trace (0..1)."""
    total = len(thinking) + len(content)
    return len(thinking) / total if total else 0.0

To prevent unbounded context bloat during long sessions, build_messages() automatically caps history to the last 6 turns while preserving system instructions.

Exact Metric Conversion (moe_demo/metrics.py)

Ollama returns internal run durations natively in nanoseconds. The metrics module transforms these numbers into actionable engineering metrics, calculating decode speed exclusively over evaluation duration (eval_duration) to avoid diluting performance stats with prefill phase latencies:

"""Inference metrics derived from Ollama's final streaming chunk plus client-side timing. Ollama reports durations in nanoseconds. All values here are *measured*, not estimated."""
from __future__ import annotations
from dataclasses import asdict, dataclass
from typing import Any
NS = 1_000_000_000.0
@dataclass
class RunMetrics:
    prompt_tokens: int = 0
    generated_tokens: int = 0 # includes reasoning ("thinking") tokens
    load_seconds: float = 0.0 # time spent loading the model into memory
    prompt_eval_seconds: float = 0.0
    generation_seconds: float = 0.0
    total_seconds: float = 0.0 # server-side total
    ttft_seconds: float = 0.0 # client-side time to first streamed token
    wall_seconds: float = 0.0 # client-side end-to-end
    @property
    def tokens_per_second(self) -> float:
        """Generation speed (decode phase)."""
        return self.generated_tokens / self.generation_seconds if self.generation_seconds > 0 else 0.0
    @property
    def prompt_tokens_per_second(self) -> float:
        return self.prompt_tokens / self.prompt_eval_seconds if self.prompt_eval_seconds > 0 else 0.0
    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.generated_tokens
    def to_dict(self) -> dict[str, Any]:
        d = asdict(self)
        d["tokens_per_second"] = round(self.tokens_per_second, 2)
        d["prompt_tokens_per_second"] = round(self.prompt_tokens_per_second, 2)
        d["total_tokens"] = self.total_tokens
        return d
def from_stats(stats: dict[str, Any], ttft: float = 0.0, wall: float = 0.0) -> RunMetrics:
    """Build RunMetrics fro
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.