Instrument Like a Learning Scientist

This is post 7 of the assessment-first series. It's about the least glamorous and most compounding part of doerkit: the telemetry that lets the platform measure itself.

Dosage is the variable that matters

Most edtech analytics report engagement - logins, time-on-page, questions attempted. Those are vanity metrics; they measure whether people showed up, not whether showing up did anything.

The variable the Dartmouth study built its whole argument on is dosage: how many lessons a student actually completed, regressed against exam performance. The distinction is the entire finding: engagement was comparable-or-higher under multiple-choice, but dosage only tracked exam scores under constructed response. If you log engagement you learn nothing; if you log dosage you learn which features work.

So doerkit logs two things from day one: an append-only events table (lesson views, quiz starts, submissions with score and pass/fail) and an attempts table (every quiz and review attempt with its score). Both carry a student key and a timestamp. That's the minimal schema, and it's enough to reconstruct dosage-versus-outcome for any cohort you later attach exam scores to.

CREATE TABLE attempts (
    id INTEGER PRIMARY KEY,
    student TEXT,
    kind TEXT, -- 'lesson' | 'review'
    lesson_id TEXT,
    score REAL,
    passed INTEGER,
    created_at TEXT
);

The dashboard is the instrument

The dosage dashboard rolls this up per student: lessons passed (the dosage number), quiz attempts, average score, reviews passed versus attempted, total events. It's a plain SQL rollup rendered as an HTML table, with no charting library and no analytics vendor.

The point isn't the visualization; it's that the raw material for an efficacy analysis exists the moment the first student touches the platform, instead of being a data-collection project you scramble to start after someone asks whether the thing works.

That framing matters for what this platform is for. Efficacy evidence is the currency of institutional edtech sales and the thing every rigorous claim in this space is missing. A platform instrumented for dosage-outcome analysis generates its own evidence base as a byproduct of being used - every cohort makes the next efficacy claim stronger. The data asset compounds; the code doesn't. That's the actual moat in this category, and it costs two database tables to start accruing.

The minimal event schema for any learning product

If you're building anything with practice and outcomes, log these from commit one, before you think you need them:

• The dose - the countable unit of work (lessons completed, problems solved), per user, timestamped. Not time-on-page.
• The verdict - pass/fail and score on each attempt, so you can separate "attempted a lot" from "attempted well."
• The retry structure - every attempt, not just the last, with its timestamp. The ~1.5-day spacing finding only existed because retries were individually logged.
• A stable subject key - so you can join to outcomes later without re-identifying anyone.

Retrofitting this after launch means the first cohort is unmeasurable, and the first cohort is exactly the one a skeptical instructor asks about. Instrument before you need it, because the need arrives as a question you can't answer retroactively.

Where this breaks

Dosage-outcome correlation is not causation, and this is the load-bearing caveat for the whole series: motivated students both complete more lessons and score higher, so raw dosage regressions are selection-inflated. The Dartmouth authors handled it by controlling for prior midterm performance, which brackets the true effect between an over-adjusted 0.71 SD and a selection-inflated 1.30 SD, and doerkit's telemetry can produce the same bracketing only if you feed it exam scores, which it doesn't collect on its own.

There's a privacy surface too: a student-keyed event log is FERPA-relevant data the moment this leaves a laptop, so the demo uses a self-chosen name and no real roster, and a genuine deployment needs a data agreement this scope deliberately avoids. Telemetry that measures learning is also telemetry that surveils learners; build it, and own that both are true.

Next post is the capstone: run the whole thing yourself, what an actual institutional deployment would still need, and an honest accounting of where the assessment-first bet holds and where it doesn't.