Small Language Models with Hugging Face transformers Library + smolLM3

Small Language Models with Hugging Face transformers Library + smolLM3 Running a 70B model in production is expensive, and for many tasks, unnecessary. If you're building a focused pipeline, a well-trained 3B model will match or beat the 70B on your specific task at a fraction of the cost. # Small But Powerful Running a 70B model in production can be expensive, slow, and, for many tasks, unnecessary. If you're building a focused pipeline like a document classifier or a multilingual support responder, a well-trained 3B model will match or beat the 70B on your specific task at a fraction of the cost. The 3B model fits entirely in a single consumer GPU. It loads in seconds. It costs nothing per token. And on constrained hardware, it's the only option that runs at all. That's the actual case for small language models (SLMs). This article uses SmolLM3, Hugging Face's flagship 3B model released on July 8, 2025, as the working model throughout. It's the most technically interesting SLM available at the 3B scale right now, trained on 11.2 trillion tokens, supporting a 128k context window, dual-mode reasoning, native tool calling, six languages, and an Apache 2.0 license with the full training blueprint published alongside the weights. The project thread woven through every section: a multilingual customer support ticket router that classifies incoming tickets by category, detects the ticket language, generates a reply in that same language, and flags low-confidence outputs for human escalation. By the end, you'll have a working pipeline you can adapt to your own domain. # Why Small Language Models Deserve More Attention The parameter-count fixation in AI is understandable but misleading. Raw scale matters, up to a point. After that point, data quality, training curriculum, and architectural choices matter more. Research from the SmolLM2 paper (arxiv, February 2025) showed that at the 1B-3B scale, carefully curated training data consistently outperforms naively scaling parameters. SmolLM3 takes that further: 11.2 trillion training tokens across a staged curriculum - web, code, math, and reasoning data - plus 140 billion reasoning tokens in post-training. The result is a model that, on zero-shot benchmarks, outperforms both Llama-3.2-3B and Qwen2.5-3B and rivals Qwen3-4B on several tasks. Take the IFEval instruction-following benchmark, where SmolLM3 scores 76.7, higher than Qwen3-4B at 68.9. On BFCL (tool calling), it ties Llama's tool-call fine-tune at 92.3. On Global MMLU (multilingual QA), it scores 53.5 against Llama-3.1-3B's 46.8. Where SLMs genuinely fall short: tasks requiring deep, broad world knowledge, competitive trivia, complex multi-hop reasoning over vast knowledge graphs, and very long-form creative writing with rich historical context. For those, you want the big model. For everything focused and domain-specific, the SLM with fine-tuning on your data will match it at a tenth of the operating cost. The Hugging Face SLM collection currently includes SmolLM3-3B (instruction-tuned, what this article uses), SmolLM3-3B-Base (untuned pretrained weights), SmolLM2-1.7B (lighter predecessor), and SmolVLM (the vision-language variant). SmolLM3 is the right choice for most new projects because dual-mode reasoning, tool calling, and the 128k context window are rare at this parameter scale. # Understanding SmolLM3's Architecture SmolLM3 is a decoder-only transformer, which is standard. Three architectural decisions inside that standard frame are less common and worth understanding because they directly affect how you deploy and tune the model. - Grouped Query Attention: Standard multi-head attention maintains separate key and value projections for each of the 16 attention heads. SmolLM3 groups those 16 heads into 4 shared query projections, reducing key-value (KV) cache memory by roughly 25% without measurable accuracy loss. This matters at inference time: a smaller KV cache means lower peak VRAM, which means you can process longer contexts or larger batches on the same hardware. - NoPE (No Positional Encoding on select layers): SmolLM3 removes rotary positional encoding (RoPE) from every fourth transformer layer, implementing a 3:1 RoPE-to-NoPE ratio. This approach comes from the 2025 paper "RoPE to NoRoPE and Back Again" and helps the model generalize over long contexts without the positional embedding degradation that affects most other small models at long sequence lengths. - Dual-mode reasoning: A single set of weights handles two modes: think andno_think . Inthink mode, the model generates a chain-of-thought trace inside ... tags before the final answer, equivalent to what separate "reasoning models" do. Inno_think mode, it answers directly. You control this per-request via the system prompt or theenable_thinking kwarg in the chat template. No extra model, no extra checkpoint. # Setting Up Your Environment Hardware minimums: | Feature | Minimum | Recommended | |---|---|---| | GPU VRAM | 6 GB (bfloat16) | 8 GB+ (RTX 3060 or better) | | System RAM | 16 GB | 32 GB | | Disk | 8 GB free | 20 GB+ SSD | | Apple Silicon | M2 8 GB | M2 Pro / M3 16 GB | CPU-only works. Expect roughly 3x slower inference for text-to-speech (TTS) synthesis and 5-8 tokens/second on generation tasks depending on your machine. Fine-tuning on CPU is impractical; use Google Colab's free T4 GPU if you don't have a local GPU. Python and packages: # Python 3.10 or newer required python --version # Create and activate a virtual environment python -m venv smollm-env source smollm-env/bin/activate # macOS / Linux smollm-env\Scripts\activate # Windows # Install all dependencies pip install \ "transformers>=4.53.0" \ "torch>=2.3.0" \ "accelerate>=0.30.0" \ "bitsandbytes>=0.43.0" \ "sentencepiece" \ "trl>=0.9.0" \ "peft>=0.11.0" \ "datasets>=2.19.0" Note: transformers>=4.53.0 is required; SmolLM3's modeling code shipped in that release. Earlier versions will fail with an unrecognized architecture error. Device detection helper (run this first): # device_check.py # Run this before anything else to confirm your setup and pick the right dtype. def detect_device(): """ Detect the best available compute device. Returns (device_str, dtype_str, load_kwargs) for use with from_pretrained. """ try: import torch except ImportError: raise RuntimeError("PyTorch not found. Install with: pip install torch") if torch.cuda.is_available(): vram_gb = torch.cuda.get_device_properties(0).total_memory / 1e9 print(f"CUDA GPU detected: {torch.cuda.get_device_name(0)} ({vram_gb:.1f} GB VRAM)") # bfloat16 is recommended for SmolLM3 -- it's the training dtype return "cuda", torch.bfloat16, {"device_map": "auto", "torch_dtype": torch.bfloat16} elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): print("Apple Silicon MPS detected") # MPS supports float16 but not all bfloat16 ops -- use float16 on Apple Silicon return "mps", torch.float16, {"device_map": "mps", "torch_dtype": torch.float16} else: print("No GPU found -- running on CPU (slower but functional)") return "cpu", torch.float32, {"device_map": "cpu", "torch_dtype": torch.float32} if name == "main": device, dtype, kwargs = detect_device() print(f"Device : {device}") print(f"Dtype : {dtype}") print(f"Kwargs : {kwargs}") How to run: python device_check.py Expected output (NVIDIA GPU example): CUDA GPU detected: NVIDIA GeForce RTX 3060 (12.0 GB VRAM) Device : cuda Dtype : torch.bfloat16 Kwargs : {'device_map': 'auto', 'torch_dtype': torch.bfloat16} # Loading SmolLM3 and Running Your First Inference With the environment confirmed, here's the complete load-and-generate pattern. This covers dtype selection, device_map="auto" for multi-GPU or CPU offload, and both thinking modes side by side. # first_inference.py # Prerequisites: transformers>=4.53.0, torch, accelerate # Run: python first_inference.py import re import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL_ID = "HuggingFaceTB/SmolLM3-3B" # ── 1. Load tokenizer and model ─────────────────────────────────────────────── print(f"Loading {MODEL_ID}...") tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) model = AutoModelForCausalLM.from_pretrained( MODEL_ID, torch_dtype=torch.bfloat16, # Match the training dtype; use float16 on Apple Silicon device_map="auto", # Spreads across all available GPUs, or CPU if none ) model.eval() print(f"Model loaded on: {model.device}") # ── 2. Generation helper ────────────────────────────────────────────────────── def generate(messages: list[dict], max_new_tokens: int = 512) -> str: """ Apply the SmolLM3 chat template, tokenize, generate, and decode. Strips the ... block from the output automatically so callers always receive the final answer only. """ # apply_chat_template formats messages using SmolLM3's built-in chat template text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) inputs = tokenizer(text, return_tensors="pt").to(model.device) with torch.no_grad(): output_ids = model.generate( **inputs, max_new_tokens=max_new_tokens, temperature=0.6, # Recommended by the SmolLM3 team for balanced output top_p=0.95, # Nucleus sampling -- keeps output focused without being repetitive do_sample=True, ) # Decode only the newly generated tokens, not the input prompt new_tokens = output_ids[0][inputs["input_ids"].shape[-1]:] raw = tokenizer.decode(new_tokens, skip_special_tokens=True) # Strip the chain-of-thought block if present. # In think mode the model prefixes its response with ... . # Callers usually only need the final answer that follows. final = re.sub(r".*? ", "", raw, flags=re.DOTALL).strip() return final # ── 3. Compare think vs no_think on the same prompt ────────────────────────── prompt = "A customer is charged twice for the same order. What are three concrete steps support should take?" # no_think: fast, direct answer -- good for high-throughput classification and replies no_think_messages = [ {"role": "system", "content": "/no_think"}, {"r