Knowledge-and-Memory-Management v0.0.2: Portable Knowledge Collection and Memory Management

Knowledge-and-Memory-Management v0.0.2 delivers a clean release with portable paths via $AGENT_HOME, replacing hardcoded personal directories. This version focuses on knowledge collection from web, video, and articles, paired with a streamlined memory management system. For developers building persistent knowledge bases, this update removes environment-specific friction and introduces unified ingestion pipelines.

Knowledge Collection

The collector module now targets three content channels with separate, configurable pipelines:

• Web: A headless browser (playwright) handles dynamic content. Users specify CSS selectors or XPath for target elements; the system extracts rendered text, discards navigation and ads, and returns clean markup. Errors from page timeouts or network failures are logged to a separate error store for retry.
• Video: Integration with yt-dlp downloads audio streams from supported platforms. Local ASR (whisper.cpp) transcribes audio to text with timestamps. Each clip is chunked by sentence boundary for manageable embedding later.
• Articles: For standard HTML pages, the collector uses a readability algorithm to isolate main content. Metadata (author, publish date, RSS tags) are parsed from JSON-LD or Open Graph headers. Fallback extraction uses boilerpipe patterns.

All collected data normalizes to a KnowledgeChunk schema: text body, source URI, content type, and a SHA256 hash for dedup. Chunks are streamed directly to the memory store without intermediate files, unless collector.cache_enabled is toggled.

Memory Management

The memory module uses a dual-layer store: Chroma for vector embeddings (using all-MiniLM-L6-v2) and DuckDB for structured metadata. On write, each KnowledgeChunk is embedded and indexed; the same hash prevents duplicate columns. Retrieval supports both semantic_search(embedding, k) and filter_search(metadata_query). Entities are extracted via spaCy NER, and a small graph of co-occurrence is built for topic clustering.

Manual cleanup is no longer needed: the MaintenanceAgent runs on schedule to purge stale chunks older than a configurable TTL (default 30 days) and rebuild indexes. The cleaned release ensures that $AGENT_HOME/data is the only runtime directory to provision.

Portable Paths and Initialization

The shift from absolute to environment-driven paths simplifies container deployments. Every internal path-including collector state, model cache, and database wal-derives from $AGENT_HOME. A simple initialization example:

from km_admin import KnowledgeManager

km = KnowledgeManager(
    base_path="$AGENT_HOME/data",
    collectors=["web", "article"]
)
km.collect("https://arxiv.org/abs/2401.12345")

At runtime, base_path expands to the environment variable. If unset, the manager falls back to ./local_data. This change also eliminates version-specific configuration files: all settings are passed as constructor arguments or dotenv overrides.

Why This Matters

Hardcoded home directories break in containers, CI runners, or when sharing configs among teams. $AGENT_HOME aligns with the XDG_CONFIG_HOME pattern and keeps the system stateless from the application perspective. For developers orchestrating multiple agents, this means one environment variable governs all storage.

The collection pipelines are now independent of the memory store. You can run collect without a running database, and the internal buffer writes to disk in JSONL format. The MemoryManager later ingests these buffers offline. This decoupling is intentional for resource-constrained environments.

Looking at the "S" in v0.0.2

The unfinished "S" in the release notes hints at the next milestone-summarization. The current KnowledgeChunk schema already includes a summary field, but the pipeline is not yet wired. In v0.0.3, we plan to plug in a local seq2seq model (e.g., bart-large-cnn) that runs on collected chunks after dedup. The memory store will then tag chunks with their summaries, enabling quick browsing without loading full text.

Conclusion

v0.0.2 is a foundation release. The portable path cleanup reduces setup time to near zero, while the modular collectors make adding new sources straightforward. Developers upgrading from earlier versions should set $AGENT_HOME and rename any old personal_path overrides in their code-everything else migrates automatically. The core loop (collect → embed → store → retrieve) is stable, and the memory management fixes from the last patch handle edge cases like concurrent chunk insertion. For now, the emphasis is on clean ingestion; real-time summarization and cross-source dedup begin in the next minor version.