Switching from GPT-5.6 Sol to GPT-6 Astra: Start with Medium Effort

GPT-6 Astra at high reasoning effort produced the implementation I decided to keep. I still plan to use medium effort by default. High handled the interactions between retries and persisted state more completely, but its implementation run took about 48 minutes against medium's 31. In a separate review of the same Sol implementation, medium found a startup failure that high missed. I ran this comparison to choose a default after GPT-5.6 Sol high. The work behind the comparison I used Galley, my tool for unattended development. It takes queued tasks through environment preparation, implementation, and verification. When a task is retried, it should preserve the agent's previous work and reuse preparation that is still valid. The four conditions were Astra low, medium, and high, plus Sol high. First, each analyzed the repository for bugs, performance problems, and maintainability improvements. I selected a common implementation scope from the Astra findings and gave all four the same plan in separate worktrees. Finally, fresh sessions reviewed identical copies of Sol's implementation. The sessions ran in Codex CLI. The agents read guidance from codex-workflows, including its coding, testing, and development-judgment skills, together with Galley's repository instructions and quality profile. The plan stated the expected behavior and constraints; the agents chose the design, order of work, and verification. These are the conditions I use for my own work. What counted as a good result In my own agent workflows, I have repeatedly seen technically defensible suggestions turn into more machinery than the product needs. I wanted to know how much of that expansion each condition could avoid without being steered through every decision. I assessed whether the agents found consequential problems, proposed sound fixes, and selected work worth maintaining. During implementation, they also had to carry the plan through to working, verified behavior without being walked through it. A small edge case could score well. Removing unnecessary path manipulation or reusing a sound lock can make a system simpler. A new recovery protocol has more to justify. I did not count findings or changed lines as a measure of quality. The analysis phase supplied a concrete example. Galley intentionally allows an operator to requeue a running task. Low and medium proposed restricting that operation. High checked the documented intent and existing test, then left it out of its formal findings. For an unattended tool with an explicit manual override, removing that override is a product decision. Tidier state transitions alone do not justify it. I checked findings against the code and tests, and credited deliberate exclusions only when the report or visible session history supported them. The evaluation notes contain the scoring criteria, reasons for the weights, findings, and full usage breakdown. A preparation step that invalidated itself The most useful implementation difference was easy to overlook in a passing test suite. Before an implementation agent starts, Galley can prepare the environment and create acceptance-test skeletons: starter tests tied to the task's requirements. If the task runs again with the same inputs, those successful preparation steps should be reusable. All four implementations added fingerprints to detect changed inputs. But Galley also writes information into the task during execution. Creating a skeleton adds an explanation to an acceptance criterion. Include that explanation in the next fingerprint, and the successful preparation step invalidates its own result. Low, medium, and Sol left a version of that problem in the setup reuse path. High separated the generated explanation from the user-supplied contract. It followed the change through preparation, task updates, and the next run. An unnecessary preparation run means another wait and another model invocation. It can also send a creator back into test files the implementation agent has already edited. Preserving valid reuse was part of the requirement, just as detecting changed inputs was. High's other choices were often modest. For large Git change sets, it extended an existing way to pass paths through standard input. For corrupt task records, it reused the existing isolation logic. It covered more behavior without inventing a separate mechanism for each case. I am taking that implementation forward in this Galley PR. It needs further review and edits. The first CI run passed on Linux and macOS but failed on Windows when a timeout test tried to clean up a log file still held open. Medium found the earlier failure Medium's implementation addressed the same plan and generally extended existing mechanisms. It left the setup reuse gap described above. At about $15.61 and 31 minutes, though, it produced a substantial change for less than high's $21.03 and 48 minutes. Then came the review phase. All four reviewed the same Sol code, so differences here describe their reviewing rather than their own implementations. Sol had added recovery for corrupt task files whose claims had expired. A claim records that a worker has taken a task. Medium followed startup and found an earlier recovery step: if a corrupt task still had an owner record from an interrupted worker, startup failed before reaching the new recovery code. Unrelated queued tasks never got a chance to run. Medium reproduced that through the actual daemon startup function. It had found a repair that could look complete locally while leaving the application unable to start. High's review caught more of the input lifecycle and another reason ordinary requeue operations invalidated preparation results. It did not catch this startup failure. For my workflow, that is a reason to keep a fresh review session even when I pay for high during implementation. Low did not save much here Low was slightly cheaper than medium during implementation, but its analysis run cost more. Across the three phases, medium was cheaper overall. Low's review did identify a data-loss path: reusing an edited input file succeeded, but a later failure while saving evidence could delete that file. Its proposed correction was small and appropriate. Roll back files created by the current attempt; preserve files brought forward from an earlier one. The review missed preparation reuse and other recovery paths. I would still try low for a small, well-defined edit. This experiment used broad repository work and does not establish how the two compare on that smaller category. Sol's review was stronger than its implementation Sol's implementation missed repeated input cleanup. It also retained both old and new scheduling paths, with some tests still exercising the old path. Future changes would have two versions of similar behavior to keep aligned. Its review found problems no Astra report included. One was another preparation-reuse failure: setup saved a learned environment profile, but its success fingerprint still described the profile from before that update. The next run rejected its own successful setup evidence. I checked this through profile persistence, reloading, and the reuse decision. Sol also caught a missing shutdown check before taking more queued work and a repository check that rejected valid paths with different capitalization on macOS. Some remedies needed more scrutiny. Galley tracks child processes in registry files. Sol correctly identified an unsafe fallback to an old shared registry, but declared new ownership metadata necessary before establishing whether the fallback could be retired with a clear migration procedure. I counted that against judgment: the problem deserved a fix, while the proposed extra state still needed a reason to exist. Sol's implementation result would have been a poor basis for predicting the value of its review. The price of a completed run These are the totals for one analysis, one implementation, and one review run per condition: | Condition | API-equivalent cost | Combined elapsed time | |---|---|---| | Astra low | $26.97 | About 49 minutes | | Astra medium | $25.67 | About 51 minutes | | Astra high | $37.23 | About 77 minutes | | Sol high | $31.79 | About 75 minutes | Medium cost about 19% less than Sol high. High cost about 17% more than Sol high, but about 45% more than medium. High's premium depends on which starting point you are considering. The implementation runs differed sharply in how many model requests they made: 238 for Sol, 80 for medium, and 107 for high. Medium accumulated about 11.1 million input tokens against Sol's 37.8 million, and generated about 50,000 output tokens against 98,000. The bill depends on those token volumes and their cache breakdown, not the request count alone. On this task, medium's lower usage more than offset Astra's higher per-token rates. These are API equivalents calculated from session usage, not charges allocated from my Codex subscription. They exclude the evaluator's work and subsequent corrections. Approval-wait overhead was estimated from the logs; the sessions ran concurrently on one machine. The resource document separates the phases and explains the calculation. What I will use next Medium is my starting point for the next broad Galley task. I would choose high up front when the work crosses retries, ownership, and persisted state, where missing a related path leaves me with another round of repair. In this implementation, high kept preparation reusable across task updates and handled recovery paths the other implementations left incomplete. This was one run per condition on one repository. The evaluation was model-assisted, and I challenged and revised assessments when their reasoning or attribution was wrong. The final scoring was not blind. I trust the concrete failures and fixes more than small differences between numerical scores. My next check is whether this choice reduces the work I have to repair afterward. I will still use a separate review p