Molodetz Scout
<!-- retoor <retoor@molodetz.nl> -->
Molodetz Scout
Molodetz Scout is a metasearch system. It queries hundreds of independent search
engines in a single pass, crawls what they surface, indexes the content into a
vector database, and returns one ranked result set. Query understanding and
result reranking are performed by a large language model; lexical (BM25) and
semantic (embedding) signals are fused with a noise penalty. Every external API
call is costed in US dollars and persisted for analysis. A live web interface
streams the entire process over WebSockets, with each search isolated in its own
worker process.
Installation
pip install molodetz-scout
Or from a checkout:
make install # pip install .
make install-dev # editable install + test dependencies
Python 3.11 or newer is required.
Configuration
Credentials are read from the environment (a .env file in the working
directory is loaded automatically). Copy .env.example to .env:
| Variable | Used by | Effect if unset |
|---|---|---|
OPENROUTER_API_KEY |
embeddings (vector retrieval) | Vector retrieval is unavailable; required for full ranking. |
DEEPSEEK_API_KEY |
query rewriting, rerank | Those stages are skipped; the pipeline still runs. |
MOLODETZ_SCOUT_DATA_DIR |
runtime data location | Defaults to the current working directory. |
Runtime data (.chroma, .costs, .users, and per-query result dumps) is
written under the data directory and is never part of the package.
Usage
Command line:
molodetz-scout-search "quantum computing"
# or
python -m molodetz_scout search "quantum computing"
Web interface (default port 17793):
molodetz-scout-web
# or
python -m molodetz_scout web
Architecture
| Component | Path | Responsibility |
|---|---|---|
| Engines | molodetz_scout/engines/ |
One self-contained script per engine; each is run as an isolated subprocess and prints JSON. |
| Orchestrator | molodetz_scout/orchestrator.py |
Discovers engines, runs them concurrently, BM25-scores and grades the output. |
| Pipeline | molodetz_scout/pipeline.py |
Crawls surfaced URLs, embeds and indexes content into ChromaDB, fuses lexical and semantic scores, applies the rerank. |
| Ranking | molodetz_scout/ranking/ |
LLM query planning, reranking, fusion, and cost accounting. |
| Web app | molodetz_scout/webapp/ |
FastAPI + Jinja + WebSocket interface, per-user history, and a spend dashboard. |
Each search runs as a server-side task that survives page navigation and socket
disconnects. Engines are executed as subprocesses, so a misbehaving engine
cannot take down the pipeline.
Engines
Engines are discovered automatically from molodetz_scout/engines/. Each engine
is a standalone script with a JSON metadata docstring, an async search_*
function, and a __main__ block that prints a JSON result object. See
CONTRIBUTING.md for the full contract and a worked example of
adding an engine.
Development
make validate # hawk: validate all source (zero errors required)
make test # integration tests
License
GPL-3.0-or-later. See LICENSE.
Comments
The noise penalty on lexical/semantic fusion is a clever guard against spammy results that rank high on both signals but lack relevance.
The per-query cost persistence is what sold me on this. Seeing exactly which engines burn money on what queries makes it trivial to cut the dead weight from your provider list.