I Built SnoozeOps: An Open-Source AI Data Janitor That Does the Boring Spreadsheet Work

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 3 - Slow and Steady I Built SnoozeOps: An Open-Source AI Data Janitor That Does the Boring Spreadsheet Work "You snooze. Your data gets cleaned." 1. The Problem: The Agony of Spreadsheet Inspection Nobody got into software engineering or data analysis because they dreamed of squinting at 5,000 spreadsheet rows looking for broken formulas, decimal slips, and negative customer tenures. Yet every week, millions of developers and analysts open CSV files and perform the same soul-draining manual inspection: - Did someone key in an order total as $1,499.00 for a USB cable that costs$14.99 ? - Did an offshore data entry clerk enter negative age ( -28 ) or future signup timestamps (2029-11-14 )? - Did an Excel formula break silently, outputting a mismatch between unit price, quantity, and total revenue? - Did an enterprise customer get assigned an NPS score of 15 when the scale strictly tops out at10 ? Missing these issues is embarrassingly easy for human eyes after 20 minutes of scrolling. But letting bad data slip into production downstream destroys analytics dashboards, corrupts machine learning training sets, and triggers costly billing reconciliations. October 20 is International Sloth Day, celebrating animals that conserve their energy and refuse to do unnecessary manual labor. That was the spark: what if an open-weight AI agent could do the tedious inspection work while you take a nap, and hand you a clean, reviewable dataset with a cryptographic audit trail when you wake up? That is SnoozeOps. 2. What I Built SnoozeOps is an open-source, local-first AI data janitor that automates spreadsheet inspection. It is not another generic chatbot where you paste a table into a prompt window; it is a dedicated developer tool that: - Profiles Your Data: Automatically computes column types, null rates, statistical moments (mean, std, median, IQR), and distribution histograms. - Detects Suspicious Flaws Multi-Tiered: Runs Median Absolute Deviation (MAD) Z-score, Tukey fences, decimal slip heuristics, domain boundary checks, Levenshtein categorical typo detection, and Prior Labs' TabPFN tabular foundation model. - Explains with Open-Weight Gemma: Generates clear, grounded explanations and proposes safe, typed corrections-backed by an explicit, transparent deterministic fallback. - Empowers Human Review: Gives engineers a clean grid interface to approve, edit, or reject proposed fixes before a single byte of data changes. - Preserves Provenance with SHA-256: Leaves raw data untouched, exports a clean CSV, and generates an itemized Markdown audit trail with cryptographic signatures. 3. Try It Yourself - 🌐 Live Demo (Render Cloud): https://snooze-ops-frontend.onrender.com - 💻 GitHub Repository: https://github.com/Megh-Shah-08/snooze-ops - ⚡ Live Backend API: https://snooze-ops-backend.onrender.com You can load built-in sample datasets (E-Commerce Retail, Clinical Health, SaaS Metrics, or the Seeded Benchmark) with one click, or drag and drop any CSV file from your computer. 4. How It Works: Architecture & Data Flow SnoozeOps separates statistical anomaly detection, neural tabular residual prediction, and natural-language explanation into dedicated, decoupled components: ┌────────────────────────────────────────────────────────┐ │ Next.js 14 Frontend │ │ Spreadsheet Grid • Anomaly Review • Settings │ └───────────────────────────┬────────────────────────────┘ │ REST JSON API ┌───────────────────────────▼────────────────────────────┐ │ FastAPI Backend │ │ │ │ ┌──────────────────┐ ┌───────────────────────────┐ │ │ │ Column Profiler │ │ Statistical Detectors │ │ │ │ (Stats & Bins) │ │ (MAD, Tukey, Slip, Typos)│ │ │ └────────┬─────────┘ └─────────────┬─────────────┘ │ │ │ │ │ │ └──────────────┬─────────────┘ │ │ │ │ │ ┌──────────────────▼──────────────────┐ │ │ │ TabPFN Tabular Prediction Engine │ │ │ │ (Prior Labs TabPFN / IsoForest) │ │ │ └──────────────────┬──────────────────┘ │ │ │ │ │ ┌──────────────────▼──────────────────┐ │ │ │ LLM Janitor & Deterministic Engine │ │ │ │ (Open-Weight Gemma 2 / Rule Fallback│ │ │ └──────────────────┬──────────────────┘ │ │ │ │ │ ┌──────────────────▼──────────────────┐ │ │ │ Audit Engine & Cryptographic Signer │ │ │ │ (Raw Preservation & SHA-256 Hashes) │ │ │ └─────────────────────────────────────┘ │ └────────────────────────────────────────────────────────┘ The Component Breakdown: - Next.js 14 Frontend: Built with TypeScript, Tailwind CSS, and Lucide icons. Uses a forest-green and warm-cream palette inspired by developer tools like Linear and Supabase. Includes an interactive spreadsheet grid that highlights anomalous cells with live tooltips. - FastAPI Engine ( backend/ ): High-performance asynchronous Python backend with NumPy and Pandas. - Statistical Engine ( backend/detectors/statistical.py ): Computes non-parametric Tukey fences ($Q_1 - 1.5 \times \text{IQR}$) and MAD Z-scores ($0.6745 \times \frac{|x - \text{median}|}{\text{MAD}}$) to remain resilient against extreme outliers that fool standard deviation tests. - TabPFN Residual Detector ( backend/detectors/tabpfn_detector.py ): Prior Labs' TabPFN tabular foundation model fits a prior-data fitted network across numeric dimensions to evaluate multivariate conditional residuals ($>3.0\sigma$). - Gemma Reasoning Engine ( backend/llm_janitor.py ): Synthesizes grounded natural-language explanations and safe corrections using Google's open-weight Gemma 2 (gemma2:2b orgemma2-9b-it ). - Audit & Cryptographic Signer ( backend/audit.py ): Applies approved transformations to a working copy, calculates input/output SHA-256 hashes, and records itemized provenance logs. 5. Key Engineering Decisions Decision 1: Never Fake Model Inference Too many AI demos hardcode fake LLM responses in the frontend. SnoozeOps takes a transparent architectural stance: - If a live Gemma model is configured (via local Ollama, Groq, or Hugging Face), it makes live structured-output calls. - If no model is configured or if the model service fails, SnoozeOps transparently falls back to a deterministic rules engine. - Every single anomaly card explicitly displays its engine origin: e.g., Engine: Gemma 2 (Live) orEngine: Deterministic Janitor Engine (Verified Fallback) . # From backend/llm_janitor.py if self.provider in ["ollama", "groq", "huggingface"] and (self.provider == "ollama" or self.api_key): try: enhanced, meta = await self._call_gemma_model(anomalies, df_context) return enhanced, meta except Exception as e: enhanced, meta = self._deterministic_fallback(anomalies) meta["is_fallback"] = True meta["fallback_reason"] = f"Model '{self.model}' unavailable: {str(e)[:150]}. Used deterministic janitor." return enhanced, meta Decision 2: Decoupling Heuristics from Language Models Language models are notoriously bad at basic floating-point arithmetic and computing column percentiles across thousands of numbers. Asking an LLM "Find the 99th percentile outlier in this 20,000-row column" is expensive, slow, and unreliable. Instead, SnoozeOps uses Python and NumPy for high-speed statistical bounds and TabPFN for tabular regression. Gemma is used for what it excels at: grounded causal reasoning and semantic formatting (e.g., explaining why a decimal was shifted and providing human-readable context). Decision 3: Sub-Millisecond Lazy Loading for TabPFN TabPFN is a powerful foundation model for tabular data, but eagerly importing PyTorch and neural network weights on application startup adds 3-5 seconds of cold-start latency. In backend/detectors/tabpfn_detector.py , I implemented lazy evaluation: _tabpfn_module = None _tabpfn_attempted = False def _get_tabpfn(): global _tabpfn_module, _tabpfn_attempted if not _tabpfn_attempted: _tabpfn_attempted = True try: from tabpfn import TabPFNRegressor _tabpfn_module = TabPFNRegressor except Exception: _tabpfn_module = None return _tabpfn_module This ensures standard statistical analysis and API health checks start in under 5 milliseconds. 6. Why Open Innovation Matters Building SnoozeOps around open-weight models and open-source tooling is not just an aesthetic choice; it fundamentally alters the software economics and privacy guarantees: - Zero Data Egress & Total Privacy: Spreadsheets often hold sensitive financial ledgers, customer emails, patient vital statistics, and private revenue metrics. Sending full CSVs to proprietary cloud APIs is an immediate compliance violation for HIPAA, GDPR, or SOC2. Because Gemma and TabPFN run locally or within your own virtual private cloud, data never leaves the host boundary. - Zero Marginal Inference Cost: Cleaning 100,000 spreadsheet rows through proprietary cloud APIs creates unpredictable token charges that scale linearly with dataset volume. With open-weight models, running 1,000 cleaning sweeps costs exactly $0 in API fees. - No Vendor Lock-In: If a proprietary provider changes pricing, deprecates an endpoint, or alters terms of service, an enterprise data pipeline breaks. Open-weight models guarantee long-term stability and reproducibility. 7. The Lazy Payoff: Measured Results & Benchmark To prove that SnoozeOps delivers measurable results rather than hand-wavy marketing, the repository includes a seeded 100-row ground-truth evaluation dataset (backend/data/ground_truth_test_data.csv ) containing 20 pre-labeled real-world anomalies. Anyone can run the live benchmark directly in the UI or via Python: python -c "import urllib.request, json; print(json.loads(urllib.request.urlopen('http://127.0.0.1:8000/api/evaluate').read())['evaluation']['comparison'])" Reproducible Benchmark Results: | Metric | Naive 3-Sigma Z-Score Baseline | SnoozeOps Hybrid AI Janitor | Gain | |---|---|---|---| | Anomaly Recall | 35.0% (Missed 13 of 20) | 100.0% (Caught 20 of 20) | +65.0% Recall | | Precision | 100.0% | 100.0% | Zero degradation | | False Positives | 0 | 0 | Clean baseline | | F1 Score | 0.52 | 1.00 | +0.48 F1 | | Runtime Latency | 2.1 m