Week 11: Welcome to my Midterm evaluation

Midterm evaluations ran from the 10th to the 14th. Deep linking and roster sync both shipped, two more pull requests merged, and then I sat down to write an honest account of where the project stands - which is how I found out that the plan I have been working from for a month builds a machine nobody can switch on. Deep linking, end to end Two pull requests, two days apart. #7772 handles the request side: a launch arrives with LtiDeepLinkingRequest as its message type instead of LtiResourceLinkRequest , and everything downstream has to route differently. The platform sends along deep_linking_settings - where to return the response, whether multiple items are allowed, whether the platform wants a title back - and that has to be stashed for the response, which happens on a completely separate request. #7777 is what the instructor actually sees: a picker listing their circuits and assignments, with the selection posted back to the LMS as a signed JWT. The satisfying part is that neither pull request needed anything new from the protocol layer. The JWT signing is the key manager from week 4. The validation on the way in is the validator from week 9. Deep linking is a different conversation with the platform, but it is the same handshake underneath, and by this point the handshake is just there. The roster #7779 reads a course roster over the Names and Roles service - pagination via Link headers, roles parsed out of the IMS role URIs. #7781 turns that into group members. The interesting decision in the second one is how users get matched. Not on email - that was week 5's lesson, and the reason for it holds here too: a platform's privacy settings may mean no email arrives at all, and even when one does, it might belong to a different CircuitVerse account than the person clicking. The match is on sub plus issuer, the same identity pair the launch signs the user in with. A roster import that mapped on email could silently add a stranger to a course. Members created this way get marked as LTI-synced, which matters for the drop-handling pull request that follows: when someone disappears from the LMS roster, only synced members get deactivated. Somebody who joined the CircuitVerse group directly should not be removed because they were never in Canvas to begin with. Two merges #7715, the JWT validator, merged on the 11th. #7763, the resource link model, merged on the 12th. Both had been sitting approved for a while - the validator since the end of July. Five merged now: the 1.1 passback fix, the deployment model, the JWKS and tool configuration endpoints, the validator, and the resource link. Eight more open and waiting. The evaluation What the evaluation actually asked me to do was describe the state of the project, which sounds trivial and was not. Writing "here is what works" forces a different question than "here is what I built this week," and the two answers were not the same shape. So I went back to the three issues this project exists to close and read them line by line against what is in the repository. Not the proposal - the issues, as filed. A pipeline nobody could feed The autograding phase of my plan had six pull requests: a test case model, a headless runner wrapper, a grading result, a job that runs on submission, a results page, and a push to the LMS gradebook. Student submits, autograder runs, grade lands in Canvas. Every one of those is about executing test cases. Not one of them is about creating test cases. The issue says it plainly - "a test case editor in the assignment settings UI where instructors can add, edit and delete test cases" - and I had somehow read that as satisfied by a database table. It is not. A model with no UI means the only way to define a test suite is the Rails console, which is not a feature an instructor can use. I had planned six pull requests that would ship a grading pipeline with no way to put anything into it. Reading further with that lens, four more gaps fell out. There was no visible/hidden flag on test cases, which both issues ask for. There was nothing for the autograde configuration - partial credit, allowed attempts, whether students see the cases before submitting. "Versioned with the assignment" had no plan behind it at all, and my model actively worked against it: one row per assignment, edited in place, so changing a suite would quietly invalidate every grade already produced from it. And the least visible one, which I only found by reading the Grade model rather than remembering it: nothing in the plan mapped a score onto an assignment's grading scale. The autograder produces a fraction. Grade validates against letter , percent , or custom , and rejects no_scale outright. Autograding an assignment with no grading scale would have failed at the last step, after the run, with the student watching. Twenty-four becomes twenty-seven Week 7's post was called "Twenty-Four Pull Requests." That plan is now twenty-seven, all three additions in the autograding phase: the test case editor, the autograde settings, and the grading-scale mapping - plus reworking the grading result to carry an attempt number and a snapshot, and folding hidden-case redaction into the results page. The snapshot is how "versioned with the assignment" gets solved, and it is the change I am happiest with. Rather than a version table with all the machinery that implies, each grading result stores a copy of the suite it ran against. Edit the suite afterwards and old results stay reproducible, because the thing that produced them is still sitting right there next to them. Three of the nine are written. The model and the runner are pushed; the editor is done and waiting locally, because it needs the model's table to exist and I would rather open one clean pull request after the first merges than one showing two commits. Reusing a format instead of inventing one The nicest thing I wrote this week is four words long. A test case needs a name, input pin values, expected output pin values, and a hidden flag. The simulator's testbench format already has groups - a named collection of cases with input and output signals. My first instinct was to add fields: a name array, a hidden array, indices lining up across them. Then: a test case is just a group with n: 1 . One case, one name, one hidden boolean. It validates against the model I had already written without changing a line of it, it invents no schema, and when the suite reaches the simulator it is a format the simulator already knows how to run. The editor serialises its table into exactly that on submit. I have written the other version of this before - the one with parallel arrays and a comment explaining how the indices correspond. It works right up until someone deletes a row from one array and not the others. Not reimplementing the simulator The runner wrapper was the piece I was most nervous about, and the design question was where the simulation actually happens. The tempting answer was Ruby: run the circuit server-side, in the language the rest of the grading code is written in. I talked myself out of it in about ten minutes. It means reimplementing thousands of lines of simulation logic, and then maintaining two engines that will drift - and the failure mode of that drift is a student's circuit passing in the browser and failing at grading. For a grading tool there is no worse bug. Driving a real browser was the next candidate, and capybara-playwright-driver is already in the Gemfile, so it looked nearly free. A browser process per submission is not nearly free. Shelling out to Node was the closest call, because there is precedent: the Yosys integration in this repository does exactly that. I went with an HTTP call to a sidecar service instead, mirroring the other half of that same integration - a URL from the environment, the same timeout-and-post shape the simulator controller already uses. It keeps the engine on the frontend repository's release cadence, where that code and the people who know it live. Every failure collapses into one error class: bad circuit data, a non-2xx, a timeout, a non-JSON body, an empty result. That is deliberate. The only caller is a background job, and there is exactly one right answer to all of them - mark the run failed, leave the student's grade alone. Where the project stands Five merged, eight open, three written and waiting. The protocol layer is done and merged. Grade passback, deep linking, and roster sync are all in the queue. Autograding is started. The plan grew by three pull requests this week, which on a burndown chart is a bad week. It is the most useful thing I did. I have spent eleven weeks getting good at making mechanisms work, and the evaluation caught me having built most of a system that no instructor could have used, because I kept reading a feature request as a description of a machine instead of a description of a person trying to get something done. Next week: the editor lands, and then the settings and the grading result behind it. Top comments (0)