RoboDoctor: A Local AI Debugging Companion for Robotics

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend What I Built Debugging robotics software has a particular kind of pain. You don't just have one terminal. You might have a ROS2 launch running in one terminal, a controller in another, a sensor node somewhere else, RViz open on the side, and a Python script running in another window. Then something breaks. You get a wall of logs, a stack trace, or a ROS2 error, and the real question becomes: "Okay... but why did it actually break?" I built RoboDoctor for a friend who works with robotics and spends far too much time chasing these errors across terminals. RoboDoctor is a local AI debugging companion that sits quietly on the desktop until something goes wrong. It can monitor multiple RoboDoctor-managed terminal sessions, detect failed commands, associate an error with the correct terminal/session, and surface the problem through a small desktop robot companion. When I double-click the robot, it takes me directly to the problematic session and starts an investigation. Instead of simply sending the error message to an LLM, RoboDoctor can gather additional diagnostic evidence through its own restricted diagnostic terminal and give that context to Gemma 3 4B running locally through Ollama. The goal is simple: Don't make the developer hunt for the error. Bring the solution to the developer. Demo Here you can watch how the whole code works! Code The complete project is open source: GitHub: https://github.com/Shreyansh08-bit/RoboDoctor.git The repository contains the backend, frontend, desktop companion, diagnostic logic, examples, tests, and startup scripts. ## How I Built It RoboDoctor is built around a simple idea: the AI shouldn't be the entire application. Gemma is the reasoning layer inside a larger debugging system. The architecture looks roughly like this: Developer Terminals ↓ Terminal Session Manager ↓ Execution + Error Detection ↓ Diagnostic Blackboard ↓ Deterministic Diagnostics ↓ Gemma 3 4B ↓ Diagnostic Investigation ↓ Diagnosis ↓ RoboDoctor Companion Multi-terminal debugging Robotics development rarely happens in one terminal. RoboDoctor therefore treats every managed terminal as an independent session. Each session tracks information such as: - command - working directory - stdout/stderr - exit code - execution sequence - ROS2 environment - current issue state This means an error in Terminal 3 does not accidentally become the diagnosis for Terminal 1. Automatic error detection The developer does not have to copy an error and paste it into a chatbot. When a command fails inside a RoboDoctor-managed terminal, RoboDoctor captures the execution context automatically. For example: ros2 run my_robot controller might return: Package 'my_robot' not found RoboDoctor associates that failure with the correct terminal session and marks it as needing attention. The desktop companion The floating robot is the entry point to the debugging workflow. It has different states depending on what is happening: - Idle - Healthy - Needs attention - Investigating - Diagnosis ready If multiple sessions have problems, RoboDoctor keeps those issues separate so the developer can choose which one to investigate. Double-clicking the companion takes the developer directly into the relevant debugging workflow. Diagnostic Blackboard Between terminal activity and the AI is a lightweight diagnostic state layer. It keeps track of: - active sessions - executions - errors - evidence - investigation state - diagnoses - confidence This prevents stale errors or example data from becoming the context for a new failure. Gemma The reasoning layer uses Gemma 3 4B, running locally through Ollama. Instead of sending terminal information to a remote API, the core reasoning loop can run locally: Developer terminal ↓ RoboDoctor ↓ Local Gemma ↓ Diagnosis Gemma receives structured debugging context rather than an unstructured dump of the entire machine. Diagnostic investigation RoboDoctor can go beyond simply repeating the error message. When the developer opens an issue, a separate diagnostic session can gather additional read-only evidence. For example, for a ROS2 problem it can investigate information such as: - installed ROS2 packages - active nodes - topics - services - ROS2 environment - package visibility - Python environment - Git state That evidence is then provided to Gemma so it can reason about the likely root cause. The diagnostic terminal is deliberately restricted. Gemma is not given unrestricted shell access, and RoboDoctor does not automatically modify source code or install packages. The AI investigates. The developer stays in control. Why Does Open Innovation Matter? For RoboDoctor, using an open-weight model locally isn't just a challenge requirement. It changes what the product can be. A robotics debugging tool can encounter information that developers may not want to send to a remote service: - source paths - terminal output - ROS2 environments - package names - hardware configuration - network information - internal project details With Gemma running locally through Ollama, the core debugging context can remain on the developer's machine. There is also no requirement for a paid cloud AI API for the core reasoning loop. The architecture becomes: local terminal → local diagnostics → local model → local diagnosis That makes a local-first debugging companion much more practical for development environments where privacy, connectivity, or API cost matters. More importantly, Gemma becomes a component that I can build a complete system around rather than the entire product. The terminal/session system, diagnostic blackboard, deterministic detectors, investigation loop, safety controls, and desktop companion are all part of the application surrounding the model. That's what open innovation made interesting for this project: The model becomes a component you can build around, rather than a service your entire product depends on. My Agent Session Prize Categories Best Use of Gemma RoboDoctor is entering the Best Use of Gemma category. Gemma 3 4B is not being used as a generic chatbot. It is the local reasoning engine inside a real debugging workflow. RoboDoctor automatically captures terminal failures, structures the evidence, investigates the relevant environment through a restricted diagnostic terminal, and provides that evidence to Gemma for root-cause reasoning. Final Thought I wanted RoboDoctor to feel less like: "Paste your error here and I'll explain it." and more like: "Something broke. I found it. Let me figure out why. Will look inside your system and give you exact solution" That's RoboDoctor. Top comments (0)