Keeping up with docs reviews: Building a tracker for Mautic's documentation PRs
Keeping up with docs reviews: Building a tracker for Mautic's documentation PRs
If you read my last post about how we keep Mautic's docs up to date with Promptless, you already know how it fits into our workflow. Within weeks of that workflow fully kicking in, there were more than 90 open documentation PRs waiting on me at once, each one needing something different: a review, a reminder, an escalation, a backport check, or a branch fix.
Don't get me wrong. Promptless worked so well, and we're thrilled with how much it helps us update our docs. But handling these PRs isn't as simple as reviewing and merging. There are several steps in between, and most of the time, I'm the one moving each PR through them because I'm currently the sole maintainer of the docs.
The review workflow
Before Promptless, code PR authors wrote their own docs updates. Now Promptless writes the first draft and opens the docs PR itself, dropping a comment on the code PR to inform that a docs PR is open and needs review. Here's how a docs PR actually moves from open to merged:
- Promptless creates a draft PR while the code PR is still open. Either the code author or the Education Team can start reviewing early, but it's not a priority yet since the code itself can still change.
- Promptless marks the draft PR as ready for review once the code PR merges. The docs PR now needs a real review, even if someone already looked at it once while it was still a draft.
- I step in to remind the author if they haven't reviewed it yet. A week of silence gets a follow-up, and after two weeks, it becomes an escalation to the Core Team.
- A Core Team member sometimes hands the docs PR back to the author. Some PRs can only be reviewed and answered by the code PR author. If the author goes quiet, it keeps showing me a reminder to nudge them again until they reply.
- The Education Team reviews the PR against our style guide once the content itself is approved. I fix grammar and wording, and make sure the new section fits well with the rest of the page.
- I approve and merge it.
With three different people potentially responsible for the same PR, and Mautic's own version branching system on top of that, there was too much to keep track of without a system.
The version branching system
Mautic keeps a branch for every version we still support, not just the newest one. Right now, that's seven branches, spanning versions 5 through 8. Both our docs repos follow the same setup - one branch per version.
Here's the part that isn't obvious from outside. A docs update doesn't belong to just one version. If something changes on an older branch, and newer branches already exist, that same change usually needs to land in every one of them. Copying the commits over like that is called a backport, and it never happens on its own. Someone has to notice it's needed and do it.
On top of that, figuring out which branch a docs PR should target isn't always obvious, since it depends on details from the code PR, rather than the docs PR itself. So every PR carries two separate questions:
- Is this on the right branch?
- Does this change still need to be copied anywhere else?
Mixing those two up is how a branch's update gets missed without anyone noticing. And once in a while, the branch a docs PR needs doesn't even exist yet. That comes down to how releases actually work on the code side.
Version 7.2 is a good example. The 7.x code branch was still being used to finish up 7.2's features. Once those were ready, the Core Team split 7.2 off into its own release branch and shipped it. After that, 7.x continued, now targeting 7.3, while 8.x was also being worked on around the same time. But neither of those branches existed yet on the docs side, so I had to create them myself before any docs PR could even point at them.
The point where I couldn't keep up
A few months into the full Promptless workflow, I had to open each PR individually to check its status. Basically, I needed to see what it needed from me next. Reminders, follow-ups, escalations, backport checklists, and wrong-branch fixes all required manual tracking, and PRs kept piling up on top of that. I wasn't only reviewing docs - I was doing manual tracking work, and it was too much to manage.
Building the tracker
I spent two days building the first working version of a tool to automate that tracking, using GitHub Actions to run it on a schedule instead of doing it manually. I've kept refining it since. It's one JavaScript file, with no dependencies. It communicates with GitHub via the REST API.
Besides that, I wanted one place where reminders to code PR authors live, so nothing gets missed and nobody's forgotten who still owes a response. I also added guidelines so a future teammate or a community member who wants to help with reviews can use and understand our docs workflow and how this tool works.
Vibe coding this didn't mean handing it over to AI and walking away. I worked through real PR scenarios one by one, tested them, hit mistakes, and rethought the structure more than once before it actually held up. Claude could write code quickly, but not every suggestion it gave me fit and aligned with the workflow, so I often pushed back. The scenario mapping, the trial and error, the actual design decisions - that part was mine.
I was strict about three things from the start:
- It's read-only, enforced by a GitHub token with read-only access. It never comments, labels, or merges anything on GitHub. I wanted the sorting handled for me, not the judgment calls.
- It never saves any todo status. It saves only raw GitHub data to go faster, and recalculates every status on every run.
- Once a week, it fetches everything and rewrites the saved data to keep it fresh.
It should be accessible, even as a small tool. Tooltips should work on keyboard focus, not just mouse hover. Links to other tabs should be announced properly. And nothing should scroll sideways or get cut off on phones. It should pass Lighthouse's accessibility checks in both light and dark mode.
How it works
I designed the tool's structure around our actual docs review workflow, shaped by the real scenarios I ran into along the way.
Pull open PRs
It fetches every open PR from our two Mautic docs repositories: user-documentation and developer-documentation-new.
Match the code PR
Docs PRs usually mention the related code PR, so the tool looks that up to check its status too. That way, the target branch and what to do next with the docs PR are accurate.
Sort by whose turn it is
Every docs PR lands in exactly one of four groups:
- Need you today. It's something only I can act on, like sending a reminder or escalating to the Core Team.
- Bring it forward. It's not urgent yet, including a new PR still needing a milestone, a review on hold because the code PR isn't merged, or a PR that's gone stale with no activity for 30+ days.
- Waiting on others or on the code PR. The code PR author or the Core Team is responsible for the next step, or the PR itself hasn't been merged yet.
- Monitoring. A normal back-and-forth is happening on the PR, and nothing needs me yet.
Check the branch
It figures out which branch the docs update should be on using the code PR's milestone and compares it to where the docs PR currently sits. If they don't match, it asks me to add a needs-rebase label so I don't forget to do that once the PR is approved. It also reminds me to add the matching docs milestone, so that stays in sync too.
Build the backport checklist
It lists every newer version branch that the same change still needs to reach, so nothing gets forgotten a version or two down the line. When a PR targets an older branch, the tool asks me to add a needs-backport label. Once the content itself is approved, I ask Promptless to open the backport PR myself. From there, the tool recognizes it and links it back to its original PR.
Track content approval status
The tool always records a formal review made through GitHub's own approve button. But if a content reviewer approves through a general comment instead, the tool can't see it unless I add the content-approved label to the PR myself. The label also matters for a formal approval, since a new commit can automatically dismiss it. Either way, once an approval is on record - formal or through the label - the tool marks the PR as ready for my final review and merge.
Catch the edge cases
- If a code PR closes without being merged, the docs PR needs to close too, and the tool marks that as one of the highest-priority actions.
- If Promptless doesn't mark a draft as ready within six hours of its code PR being merged, the tool flags it for me to change manually.
- If someone outside the team leaves a comment that never got a reply, it surfaces that as well.
Build the dashboard
It builds one self-contained HTML page, dark mode included, filterable by repo and priority. A separate priority filter - Critical, Serious, Act, Triage, or Stale - narrows things down further. There's also a search box, checkboxes to mark once I've finished a task, and a switch to hide anything already checked off.
Build the reminders page
It builds a separate page listing which code PR authors still need to respond, grouped by person, plus what's waiting on the Core Team. Once an author actually responds, their name drops off the page on its own. I still remind each author individually, but I also drop this page in Slack weekly so everyone can see their own list of PRs to review in one place.
Build the guide
It also publishes a short guide page explaining what everything on the dashboard means, so anyone using it can understand a label or a status without having to ask.
Run automatically
A GitHub Actions workflow fetches data roughly every hour on weekdays and publishes it as a website hosted on GitHub Pages.
What I learned building this
Building this taught me things about our releases and branching I still didn't fully understand, even after more than a year maintaining this project. Every time I thought I finally got it, I hadn't. Well, at least not completely. It took building this tool and asking our Project Lead and Core Team more questions before it actually clicked.
For this tool to be accurate, the right docs branch depends on the code milestone, not the branch name. I'd misunderstood it before. I thought a code PR targeting something like 7.x should point the docs update to the latest existing 7 branch, but that's not the rule. The real answer is the milestone on the code PR. My first version of the tracker assumed the newest-branch guess, and it caused the tool to flag at least one real PR incorrectly.
New scenarios keep showing up, and I have to move fast when they do. Not every workflow situation is a regular one. Every so often something comes up that I hadn't planned for, and because other people rely on this tool too, I can't leave it broken. I have to notice it and fix it quickly.
Not every PR can be top priority, and that's okay. Treating every open PR as equally urgent was never sustainable. Sorting by real priority means some things wait, on purpose, and that's not a failure. It's the only way to actually keep up.
Clear guidelines matter. Whoever picks it up next - a future teammate or a code PR author trying to understand what a label means - needs to be able to follow it without me standing there to explain. That's a big part of why the tool publishes its own guide page alongside the dashboard.
Ask AI to remember its own recurring mistakes. A couple of times, Claude linked the code PR right in the docs PR description, which tagged that PR and pulled in notifications that had nothing to do with it. Another time, it kept committing changes without asking, even within the same session, when I like to test things out first. Once I noticed the pattern, I told Claude to note both in its memory: don't link the code PR in the description, and don't commit anything without my permission first.
Final words
Promptless solved the problem of who writes the docs. This tracker solved the next problem right behind it - keeping track of it all, across every version we support.
adiati98 / mautic-docs-prs-tracker - Mautic User and Developer documentation PRs tracker and reminder for Mautic Education Team
Mautic Docs PR Tracker 📊
Note 👀 Just here to review docs PRs? You don't need to set anything up. Open the Docs PR Tracker dashboard and read the tracker guide to learn how to use it.
This README is for people setting up or maintaining the tool itself. A local Node.js script that tracks open Mautic docs PRs and generates a dashboard showing what needs attention. The script fetches open PRs from Mautic's user documentation and developer documentation repos. Most docs PRs link to a code PR they document. Some are standalone. Either way, the script works out whose turn it is to act - the reviewer, the code author, or nobody right now.
This helps the Education Team, and anyone who wants to help review docs PRs, know what to do next. It writes three HTML files: tracker-report.html - the main dashboard, tracker-reminders.html - a page you can…
If you're running an AI-assisted docs workflow and starting to feel as overwhelmed as I was, I hope this gives you somewhere to start. What part of your work as an open source maintainer takes up the most of your time without you realizing it? I'd love to hear about it!
Comments
No comments yet. Start the discussion.