BrighTO-Router: an open-source Rust LLM gateway for fast, self-hosted AI traffic

Overview

BrighTO-Router is a free, open-source LLM gateway and router written in Rust. Its goal is to give a team a single self-hosted API endpoint for AI traffic while keeping the hot path small, fast, and understandable.

Features

• OpenAI-compatible chat, completions, and embeddings
• Anthropic Messages routing
• Local/custom OpenAI-compatible endpoints
• Rerank adapters
• ASR/transcription proxying
• Team API keys and token budgets
• PostgreSQL usage metadata ledger
• Round-robin and weighted Model Groups for model load balancing
• One Docker install

Why Rust

LLM gateways sit directly on the request path, so the router should add as little latency and operational complexity as possible. Rust provides predictable memory use, fast async networking, and a deployment model that fits a single Docker image.

Benchmarks

The project publishes deterministic mock-backend benchmarks up to 1M-token payloads. These measurements report router overhead, not model inference speed, so the results are repeatable and honest.

Links

• Repo: https://github.com/thusinh1969/BrighTO_Router
• Docs: https://thusinh1969.github.io/BrighTO_Router/
• Docker: thusinh1969/brighto_airouter:v1