How to Build an AI Agent That Fixes Bugs Reliably: Practice in Sharkly

At Sharkly, people rarely fix routine bugs by hand. Most bugs go directly to an AI agent.

Sharkly is the product our team is building, and we use it to run our own product and engineering workflow. Our dedicated agent for bug fixes and small engineering tasks is called Sherlock. It can discuss an issue, inspect the relevant repository, trace the implementation, run tests in a real development environment, and verify UI changes in a browser. When visual behavior changes, it also captures screenshots as evidence.

According to our internal data, Sherlock completed 13% of all Sharkly tasks over a recent 15-day period. Of the latest 100 tasks assigned to it, 90 were completed successfully.

Sherlock runs on a shared development Computer using the oh-my-pi Runtime and the GPT 6 Astra model. We assign it dozens of bugs and small improvements every day. More importantly, we trust it to handle them without constant supervision. That trust does not come from assuming the model is always right. It comes from putting the agent inside a strict, inspectable workflow.

A Real Bug Fix from Start to Finish

Consider a recent issue in Sharkly. The temporary-directory list in the Clear Build Cache dialog could grow beyond the available space, but the list was not scrollable. Oliver discovered the problem and posted it in our internal Slack channel. Emma then mentioned the Sharkly Task Assistant and asked it to turn Oliver’s report into a structured task.

Emma assigned the new task to Sherlock. Sherlock spent 14 minutes reproducing the issue, tracing the implementation, and locating the root cause. Sherlock then presented three implementation decisions that required human input before development could begin.

Emma selected the recommended options and sent the explicit approval token: 1A 2A 3A [START DEVELOPMENT]

Only then did Sherlock begin changing the product code. After another 37 minutes, it reported that implementation and verification were complete. It provided three concrete items for review:

• A code pull request
• A preview URL
• Screenshots of the corrected behavior

Emma reviewed the change and its evidence. Once the fix was confirmed, Emma sent the second approval token: [SUBMIT FOR TESTING]

That token authorized Sherlock to merge the change into the development branch and move the task into the automated testing workflow.

Emma made decisions at only two points:

• Approve the proposed solution and allow product code changes.
• Approve the completed change and allow it to enter the testing pipeline.

Everything between those gates was handled autonomously. There was no need for a long back-and-forth conversation about every file, command, or implementation detail. That is how one engineer can supervise dozens of small fixes without becoming the bottleneck.

The Platform Behind Sherlock

Sharkly is the team-level project management and execution layer for AI coding agents. People plan work in Tasks and Sprints, assign it to Agents or Crews, and run those agents safely in parallel on their own Computers.

Sharkly separates an Agent from its Runtime. The Agent carries its identity, instructions, Skills, and operating rules. The Runtime is the coding tool that performs the work on a selected Computer, such as Claude Code, Codex, or another compatible tool. This separation lets a team choose its execution tools while keeping task context, approvals, run history, and evidence in one shared workspace.

Sharkly also connects Agents to the team’s operating process. Automations can start work when a task changes status, while explicit approval points keep high-impact actions under human control. Agents research, execute, test, and report. People set direction, grant authority, and accept the result.

For Sherlock, Sharkly provides the task, conversation, execution history, and approval gates. A shared development Computer provides the repository, dependencies, browser, and other tools, while the configured Runtime executes Sherlock’s instructions.

Sherlock’s 10-Step Workflow

Sherlock follows a fixed workflow with ten stages. It cannot skip ahead simply because a solution looks obvious.

1. Initialize the Task

Sherlock checks the task ID, title, description, labels, and priority. It confirms that it is running inside the task workspace provided by Sharkly and selects the correct baseline branch. An urgent task might use main. A high-priority task might use staging. Other tasks normally follow the repository’s default integration branch.

2. Diagnose the Problem

The diagnosis stage is limited to reproducing the issue, collecting evidence, and identifying the root cause. Sherlock is not allowed to modify product code during this stage, even when the fix appears straightforward. Its findings must be written to diagnosis.md. Every later proposal must be traceable back to evidence recorded in that file.

3. Draft a Solution

After identifying the root cause, Sherlock creates a proposed solution. It explains what should change, which behaviors must remain unchanged, and which unresolved decisions require human input. If the change affects user-facing copy, Sherlock first reads the repository’s terminology and localization rules. It then prepares several candidate phrasings instead of inventing text without context. The proposal is stored in idea.md and refined as feedback arrives.

4. Wait for Development Approval

This is the first human gate. Sherlock may understand the problem and have a complete solution, but it still does not have permission to change product code. It must wait for the exact [START DEVELOPMENT] token. Messages such as “continue,” “looks good,” or “go ahead” do not unlock the next stage. Natural language is too easy to misinterpret when authorization matters.

5. Explore and Implement

After approval, Sherlock performs a focused implementation exploration. It reads the relevant files, traces call sites, checks existing tests, and learns the repository’s established conventions. The exploration produces findings.md and plan.md. Sherlock then implements the approved solution. Any necessary departure from the plan is documented in plan-delta.md.

6. Run a Clean-Room Review

Finishing the code does not mean the task is complete. Sherlock first synchronizes with the authoritative baseline branch. It then performs a clean-room review, temporarily setting aside the assumptions it used during implementation and evaluating the change as an independent reviewer would.

The review asks questions such as:

• Does the change address the actual root cause?
• Have all affected call sites been migrated?
• Do the tests cover the meaningful risks?
• Does any obsolete code path remain?
• Did the implementation introduce unrelated changes?

7. Patch Blocking Issues

If the review finds a blocking problem, Sherlock enters a patch stage. It fixes the issue, synchronizes with the baseline again, and repeats the clean-room review. This loop can run at most six times. If blocking issues remain after the sixth review, the automated workflow stops and requests human intervention. The limit prevents the agent from running indefinitely in a cycle where it writes, reviews, and repeatedly repairs its own code.

8. Verify Real Behavior

After the review passes, Sherlock runs the repository’s declared test suite. It also verifies the behavior that actually changed. A successful test command is only one piece of evidence. UI changes require proof from the real interface. Backend changes need evidence from APIs, logs, queries, or executed commands that reflect runtime behavior. The verification standard is simple: show that the feature works, not merely that a command exited with status code zero.

9. Wait for Testing Approval

Sherlock assembles a testing package containing the implementation summary, code diff, automated decisions, test results, preview links, and visual or runtime evidence. It then stops at the second human gate and waits for [SUBMIT FOR TESTING]. This token authorizes Sherlock to create the final commit, push the code, perform the target-branch operation, and update the Sharkly task status.

10. Submit the Change for Testing

Finally, Sherlock merges the change into the development branch and moves the task to Ready for Testing. A testing agent can then continue the workflow.

The Core Design: A State Machine, Not a Persona

Sherlock’s instructions are not a conventional role prompt that says, “You are an expert software engineer.” They define an engineered, single-agent defect-resolution state machine. The result is closer to an executable development procedure than a chatbot personality. Sherlock can work independently for most of the task, but it cannot decide when to start modifying product code. It also cannot decide when to push or merge the result. Those transitions belong to people.

Every stage is recorded in a persistent state.json file. Sherlock must read and validate that file before taking action, which makes the current state explicit and recoverable.

The design has some important parts.

1. A Narrow Role and Clear Boundaries

Sherlock is allowed to work only on the Sharkly task currently assigned to it. It may not search for unrelated work, create replacement tasks, delegate the task to another agent, or perform actions that belong to a later stage. These constraints reduce ambiguity. The agent does not need to decide what kind of work it should do. It only needs to complete the current task within the permitted workflow.

2. An Explicit Order of Authority

When information conflicts, Sherlock follows a defined priority order:

1. Agent instructions and explicit human decisions
2. Repository-level guidance such as AGENTS.md, documentation, and branch policies
3. The task description and approved proposal
4. The source code and observed runtime evidence

The important principle is that repository facts override generic model knowledge. A model may remember how a framework is usually configured, but that does not prove that the current repository follows the same convention. If the repository declares its own test commands or branch policy, Sherlock must follow them. When documentation, code, and runtime behavior disagree, Sherlock must expose the conflict. It cannot silently choose the most convenient interpretation.

3. Persistent State

After initialization, state.json is the single source of truth for the workflow’s current stage.

The file records:

• Task information
• The active stage
• Pending human approvals
• Approved decisions
• Stage artifacts
• Verification status
• Transition history

Every valid state transition records the previous stage, the next stage, the reason for the transition, and the relevant files and hashes. A transition that requires approval also stores the approval token and the exact version of the proposal that was approved.

This persistence allows Sherlock to recover after an interrupted session. A new run does not guess what happened from chat history. It reloads the state and stage artifacts from disk. If state.json is missing, corrupted, or internally inconsistent, Sherlock stops and reports a state error. It cannot manually advance the stage or reconstruct a missing approval from an informal chat message.

4. Two Human Approval Gates

The two approval tokens control two different categories of high-impact action. [START DEVELOPMENT] authorizes writes to product code. It confirms that a person has reviewed and accepted the proposed solution. [SUBMIT FOR TESTING] authorizes commits, pushes, target-branch operations, and task-status changes. It confirms that a person has reviewed the implementation and its evidence.

This divides risk into two parts:

• The risk of changing the code
• The risk of sending that change into the team’s shared delivery process

Sherlock can complete a large amount of autonomous work between the gates, but it cannot cross either boundary by itself.

5. Required Stage Artifacts

Each stage must produce a concrete artifact: diagnosis.md explains how the issue was reproduced, identifies the root cause, and records the supporting evidence. idea.md describes the proposed solution and the choices approved by the user. findings.md records implementation exploration. plan.md explains what will change and how the change will be verified. plan-delta.md documents deviations from the approved plan. review-report.md contains the clean-room review results. ship-summary.md contains the final delivery and verification information.

Sherlock cannot claim that a stage is complete when its required artifact is missing. A chat message saying “diagnosis complete” has no authority by itself. The state transition is valid only when both the state record and the required artifact exist.

6. Stage-Specific Skills

Sherlock uses seven specialized Skills in addition to its main instructions:

• lattice-diagnose
• lattice-idea
• lattice-explore
• lattice-implement
• lattice-review
• lattice-verify
• lattice-deliver

The main instruction set controls stages, permissions, and transition rules. Each Skill defines how work should be performed inside a specific stage. Before entering a stage, Sherlock must read the relevant Skill in full and record that action in the state history. The resulting artifact must also demonstrate that the Skill was followed. Recording “Skill read” without producing compliant work is not enough. This separation keeps the central state machine focused while allowing each stage to have detailed, testable operating rules.

7. Evidence-Based Verification and Delivery

The verification stage has a deliberately high standard of evidence. Passing tests does not prove that the complete feature works in its real environment. A user-visible change must be checked in the actual interface. Backend changes must be supported by API responses, logs, queries, or command output. Before submission, Sherlock pushes its working branch to a dedicated remote review branch and provides a comparison link. The reviewer can inspect both the code diff and the associated verification evidence.

8. A Consistent Response