Inside NVIDIA Nemotron 3.5 Lightning: Building a (Local) Mixture-of-Experts Explorer
Introduction
Modern AI systems face a fundamental tension: users demand the deep reasoning and multi-step planning of large frontier models, yet running dense 30B+ parameter architectures for high-frequency subtasks (like code routing, tool parsing, and syntax validation) imposes massive computational and financial overhead.
To solve this, NVIDIA introduced Nemotron 3.5 Lightning, an open-weight 30-billion parameter Mixture-of-Experts (MoE) model that activates only 3 billion parameters per token.
This article explores nvidia-nemotron-lightning-35-test, a production-grade local explorer and testing framework designed to interact with, monitor, and visualize Nemotron 3.5 Lightning locally via Ollama.
Rather than just wrapping API endpoints, this project provides a clear architectural blueprint for building transparent, low-latency AI applications.
Implementation
The nvidia-nemotron-lightning-35-test project provides a robust framework and sandbox to explore, serve, and evaluate NVIDIA Nemotron 3.5 Lightning.
The repository architecture is structured around core demonstration modules, documentation sets, and utility scripts designed to test MoE behavior, speculative decoding, and application-level routing.
At the heart of the system is a Mixture-of-Experts (MoE) transformer backbone. Unlike dense models where all parameters are activated for every token, Nemotron 3.5 Lightning routes tokens through sparse expert layers, ensuring high representational capacity (30B total) combined with rapid execution speed (3B active parameters).
Architectural Philosophy: The MoE Paradigm & Efficiency
Unlike dense models that utilize their entire parameter set for every token generated, Nemotron 3.5 Lightning routes tokens dynamically through sparse expert layers.
This decoupled design provides the representational capacity of a 30B model with the execution speed of a 3B model.
+-------------------------------------------------------------------+
| Client / User Input |
+-------------------------------------------------------------------+
β βΌ
+-------------------------------------------------------------------+
| NeMo Switchyard / Router Layer |
+-------------------------------------------------------------------+
β βΌ
+-------------------------------------------------------------------+
| NVIDIA Nemotron 3.5 Lightning (MoE) |
| - 30B Total Parameters / 3B Active Parameters |
| - Multi-Token Prediction (MTP) & Speculative Decoding Support |
+-------------------------------------------------------------------+
β β βΌ βΌ
+-------------------------+ +-------------------------+
| Draft Models | | Serving Frameworks |
| (DSpark / MTP / DFlash) | | (vLLM / llama.cpp / LoRA)|
+-------------------------+ +-------------------------+
To maintain complete intellectual honesty, the projectβs application architecture enforces a strict boundary between hard telemetry (data directly measured from backend inference) and illustrative modeling (educational visual simulations):
| Data Component | Origin | Status |
|---|---|---|
| Connection State & Model Metadata | Ollama /api/version, /api/tags, /api/ps blog-post-V2.md |
Measured blog-post-V2.md |
| Generation Speed & Token Counts | Ollama final streaming chunk (eval_count, eval_duration) blog-post-V2.md |
Measured blog-post-V2.md |
| Time to First Token (TTFT) | Client-side performance timer blog-post-V2.md | Measured blog-post-V2.md |
| Expert Activation Heatmap | moe_demo/router_sim.py (Deterministic BLAKE2b simulation) blog-post-V2.md+ 1 |
Illustrative blog-post-V2.md |
β οΈ Design Note on Ollama: Ollama does not currently expose internal per-token router telemetry (e.g., revealing exactly which expert ID fired for a specific token). The application explicitly flags its expert heatmaps as simulated, preserving user trust and technical accuracy.
Core Project Structure
The codebase is organized around a clean separation of concerns, isolating the GUI layer from core business logic and state computation:
nvidia-nemotron-lightning-35-test/
βββ Docs/ # System documentation and architectural references
βββ docs_tools/ # Automated document parsers and build scripts
βββ moe_demo/ # Core application packages
β βββ __init__.py
β βββ categories.py # Task heuristic classifiers and expert affinities
β βββ config.py # Environment-based configuration loader
β βββ engine.py # UI-agnostic request builder and stream handler
β βββ metrics.py # Nanosecond-to-throughput converters
β βββ ollama_client.py # Robust REST wrapper for Ollama APIs
β βββ router_sim.py # Deterministic top-6-of-128 routing simulator
β βββ ui_components.py # Streamlit visual cards and heatmaps
βββ app.py # Central Streamlit GUI entrypoint
Implementation Deep Dive
Environment Configuration (moe_demo/config.py)
Configuration is managed via environment variables with fallback support for local .env files. Using a frozen dataclass ensures settings remain immutable throughout the application lifecycle:
"""Runtime configuration. Values come from the environment / `.env` (see `.env.example`)."""
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
def load_env_file(path: Path | None = None) -> None:
"""Load KEY=VALUE lines from `.env` into os.environ without overriding real env vars. A tiny parser is used on purpose so the app has no extra dependency and also works when started with `streamlit run app.py` instead of `scripts/start.sh`."""
path = path or ROOT / ".env"
if not path.is_file():
return
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
value = value.strip().strip('"').strip("'")
os.environ.setdefault(key.strip(), value)
@dataclass(frozen=True)
class Settings:
base_url: str
model: str
keep_alive: str
request_timeout: float
def get_settings() -> Settings:
"""Build settings from the environment (defaults only for non-sensitive values)."""
load_env_file()
return Settings(
base_url=os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434").rstrip("/"),
model=os.environ.get("OLLAMA_MODEL", "nemotron-3.5-lightning:30b-a3b-mlx"),
keep_alive=os.environ.get("OLLAMA_KEEP_ALIVE", "15m"),
request_timeout=float(os.environ.get("OLLAMA_TIMEOUT", "600")),
)
Note: The default OLLAMA_TIMEOUT is set to 600 seconds to gracefully handle cold-starting a massive βΌ19 GBβΌ19 GB model artifact without throwing premature timeouts.
Resilient Ollama Client (moe_demo/ollama_client.py)
The client wraps Ollama's REST API (/api/version, /api/tags, /api/ps, /api/generate, and /api/chat), translating raw HTTP errors into a clean exception hierarchy:
class OllamaError(Exception): ...
class OllamaConnectionError(OllamaError): ...
class OllamaModelNotFound(OllamaError): ...
class OllamaTimeout(OllamaError): ...
During streaming chat completions, NDJSON response streams are safely parsed into typed StreamEvent objects, separating thinking/reasoning blocks (thinking) from final generation tokens (content) and server statistics (done):
for line in resp.iter_lines():
if not line:
continue
data = json.loads(line)
if "error" in data:
raise OllamaError(str(data["error"]))
msg = data.get("message") or {}
if msg.get("thinking"):
yield StreamEvent("thinking", msg["thinking"])
if msg.get("content"):
yield StreamEvent("content", msg["content"])
if data.get("done"):
yield StreamEvent("done", stats=data)
UI-Agnostic Generation Engine (moe_demo/engine.py)
The engine converts user parameters into validated option payloads and manages conversation window constraints:
"""UI-independent generation logic: parameters -> Ollama request -> live metrics. Keeping this separate from Streamlit makes it unit-testable against a stub Ollama server."""
from __future__ import annotations
import time
from dataclasses import dataclass
from typing import Iterator
from .metrics import RunMetrics, from_stats
from .ollama_client import OllamaClient
THINK_MODES = ("Auto (per category)", "Off", "On", "Medium")
def build_options(temperature: float, top_p: float, num_ctx: int, num_predict: int) -> dict:
"""Map UI values to Ollama `options`. These are re-read on every request, so changing a slider affects the very next message."""
if not 0.0 <= temperature <= 2.0:
raise ValueError("temperature must be within 0..2")
if not 0.0 < top_p <= 1.0:
raise ValueError("top_p must be within (0, 1]")
if num_ctx < 512:
raise ValueError("num_ctx must be at least 512")
return {"temperature": temperature, "top_p": top_p, "num_ctx": int(num_ctx), "num_predict": int(num_predict)}
def resolve_think(mode: str, category_think: str) -> bool | str:
"""Translate the UI reasoning selector into the value of Ollama's `think` field. The model accepts False, True or "medium" (see `ollama show`)."""
chosen = category_think if mode.startswith("Auto") else mode.lower()
if chosen == "off":
return False
if chosen == "medium":
return "medium"
if chosen == "on":
return True
raise ValueError(f"unknown reasoning mode: {mode!r}")
def build_messages(system_prompt: str, history: list[dict], prompt: str, use_history: bool, max_turns: int = 6) -> list[dict]:
"""System prompt + (optionally) the last `max_turns` turns of the conversation + new prompt."""
messages: list[dict] = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
if use_history:
for m in history[-max_turns * 2:]:
if m["role"] in ("user", "assistant") and m.get("content"):
messages.append({"role": m["role"], "content": m["content"]})
messages.append({"role": "user", "content": prompt})
return messages
@dataclass
class LiveUpdate:
"""Snapshot yielded while streaming."""
content: str
thinking: str
chunks: int # streamed chunks ~ tokens (live approximation)
elapsed: float
ttft: float
live_tps: float # approximate until the final exact metrics arrive
final: RunMetrics | None = None
def stream_answer(client: OllamaClient, model: str, messages: list[dict], options: dict, think: bool | str, keep_alive: str) -> Iterator[LiveUpdate]:
"""Stream a reply and yield LiveUpdate objects; the last one carries exact `final` metrics."""
start = time.perf_counter()
first: float | None = None
content, thinking, chunks = "", "", 0
for ev in client.chat_stream(model, messages, options=options, think=think, keep_alive=keep_alive):
now = time.perf_counter()
if ev.kind in ("content", "thinking"):
if first is None:
first = now - start
chunks += 1
if ev.kind == "content":
content += ev.text
else:
thinking += ev.text
gen_time = max(now - start - (first or 0), 1e-6)
yield LiveUpdate(content, thinking, chunks, now - start, first or 0.0, chunks / gen_time if chunks > 1 else 0.0)
elif ev.kind == "done":
wall = now - start
final = from_stats(ev.stats, ttft=first or wall, wall=wall)
yield LiveUpdate(content, thinking, chunks, wall, first or wall, final.tokens_per_second, final)
def reasoning_share(thinking: str, content: str) -> float:
"""Fraction of streamed characters that belong to the reasoning trace (0..1)."""
total = len(thinking) + len(content)
return len(thinking) / total if total else 0.0
To prevent unbounded context bloat during long sessions, build_messages() automatically caps history to the last 6 turns while preserving system instructions.
Exact Metric Conversion (moe_demo/metrics.py)
Ollama returns internal run durations natively in nanoseconds. The metrics module transforms these numbers into actionable engineering metrics, calculating decode speed exclusively over evaluation duration (eval_duration) to avoid diluting performance stats with prefill phase latencies:
"""Inference metrics derived from Ollama's final streaming chunk plus client-side timing. Ollama reports durations in nanoseconds. All values here are *measured*, not estimated."""
from __future__ import annotations
from dataclasses import asdict, dataclass
from typing import Any
NS = 1_000_000_000.0
@dataclass
class RunMetrics:
prompt_tokens: int = 0
generated_tokens: int = 0 # includes reasoning ("thinking") tokens
load_seconds: float = 0.0 # time spent loading the model into memory
prompt_eval_seconds: float = 0.0
generation_seconds: float = 0.0
total_seconds: float = 0.0 # server-side total
ttft_seconds: float = 0.0 # client-side time to first streamed token
wall_seconds: float = 0.0 # client-side end-to-end
@property
def tokens_per_second(self) -> float:
"""Generation speed (decode phase)."""
return self.generated_tokens / self.generation_seconds if self.generation_seconds > 0 else 0.0
@property
def prompt_tokens_per_second(self) -> float:
return self.prompt_tokens / self.prompt_eval_seconds if self.prompt_eval_seconds > 0 else 0.0
@property
def total_tokens(self) -> int:
return self.prompt_tokens + self.generated_tokens
def to_dict(self) -> dict[str, Any]:
d = asdict(self)
d["tokens_per_second"] = round(self.tokens_per_second, 2)
d["prompt_tokens_per_second"] = round(self.prompt_tokens_per_second, 2)
d["total_tokens"] = self.total_tokens
return d
def from_stats(stats: dict[str, Any], ttft: float = 0.0, wall: float = 0.0) -> RunMetrics:
"""Build RunMetrics fro
Comments
No comments yet. Start the discussion.