How I Built a File-Timestamp-Based Feedback Loop to Enforce AI Output Quality

The problem: AI outputs are probabilistic, and prompts have a ceiling

LLMs produce probabilistic outputs. No matter how good your prompt is, edge cases will fail - hallucinations, omissions, format drift, and confident-sounding rationalizations that don't hold up.

I noticed this while using Claude Code daily: the AI would say "done" but the file wasn't written. It would claim "logs updated" but the timestamps were three days old. The AI wasn't lying - probabilistic output is inherently unstable.

Pure prompt engineering is fighting probability with probability. The ultimate defense must be deterministic, mechanical checks.

The solution: 4 of 5 steps are scripts. Only 1 requires AI.

I built an agent configuration system with a closed-loop feedback mechanism:

• self-model.md (current self-cognition)
• ↓ Session executes (AI works based on config)
• ↓ Growth data accumulates (what worked, what failed)
• ↓ quality-gate.py detects staleness (file timestamps + exit codes, pure Python)
• ↓ Writes .self-model-stale flag to disk
• ↓ Next startup: health-check.py detects flag → triggers AI to regenerate self-model
• ↓ (loop closes)

4 steps are mechanical scripts: file timestamp checks, exit code gates, JSONL audit trails, flag file I/O. 1 step requires AI: content regeneration - synthesizing accumulated growth data into updated self-cognition.

Machines do the checking. Humans and AI do the judging. This isn't philosophy - it's engineering.

Key design decisions

1. Zero dependencies, stdlib only
   Every script uses only Python's standard library. A quality check tool can't introduce new dependency risks.

2. Dual-layer gate: soft reminder + hard block

   • Process layer (soft): Rule execution rate low? Remind, but don't block.
   • Output layer (hard): Learning logs not updated? Exit 2, hard block.
     Delivery must be complete. The boundary isn't importance - it's "can this be fixed later?"

3. Filesystem as database
   No vector databases. No cloud services. All identity data, growth logs, and audit records are local Markdown + JSON files. Git-auditable, offline-capable, fully self-sovereign.

External validation: submitting to a 100K-star project

I extracted one module (delivery-gate) from my personal system and submitted it to ECC (100K+ stars). Result: maintainer daltino reviewed and approved it with praise. Maintainer affaan-m personally merged two follow-up PRs.

A 200-line Python script went through 4 rounds of community bot review + human maintainer review, catching 9 issues I hadn't found in self-testing. Open-source community review is the best free QA you'll ever get.

This became my "open-source flywheel" methodology: build for yourself → extract module → find a community gap → submit PR → merge back into your own system.

If you want to do something similar

• Dogfood it first. My system ran through 50+ real sessions before I submitted anything.
• Scripts, not prompts. If you can check it with an if/else in Python, don't describe it in natural language.
• Small PRs win. For large projects, 100-300 lines is the sweet spot for maintainer review.
• Use the gap-filling template. "This repo has X and Y. But there is no Z. This PR fills that gap."

The real takeaway

I'm a junior-year undergrad. My raw coding speed probably doesn't beat CS majors who eat LeetCode for breakfast. But I've learned one thing that matters more:

The core competency of the AI era isn't typing speed - it's knowing what to let AI do, and what must be enforced with deterministic rules.