Profiling a Python code analysis pipeline: 157s → 12s by chasing bottlenecks

I recently spent some time profiling a Python code analysis pipeline that processes large repositories into a structured knowledge bundle. Rather than trying random optimizations, I profiled each stage, fixed the biggest bottleneck, and then profiled again. What surprised me was that every optimization exposed a completely different bottleneck.

Benchmark Results

The benchmark was run on a real workspace containing: 23 GB source tree, ~18,000 source files, and ~41,000 extracted concepts.

Key Optimizations

The biggest improvements came from removing unnecessary work rather than simply adding more parallelism:

• Replaced Path.rglob() with os.walk() and directory pruning (avoiding hundreds of thousands of unnecessary filesystem visits).
• Parallelized parsing with ProcessPoolExecutor because parsing was genuinely CPU-bound.
• Replaced repeated list scans with precomputed indexes, caching, and set-based lookups in the linker.
• Profiled inside the write stage and discovered that yaml.safe_dump() accounted for nearly 95% of render time for a fixed front matter schema. Replacing it with a schema-specific serializer became one of the biggest remaining wins.

Takeaways

One takeaway that surprised me is how misleading optimization can be without profiling. I initially assumed parsing would dominate, but after each fix, a completely different stage became the bottleneck.

Resources and Feedback

I documented the entire optimization journey-including benchmark methodology, implementation details, trade-offs, and the optimizations that didn't matter-in this write-up: Performance write-up: performance

The implementation is part of an open-source project called OKF Generator , but I tried to make the article useful even if you're not interested in the project itself: okf-generator

I'd love feedback from anyone who has worked on compilers, language servers, static analysis tools, code indexers, or other large-scale repository processing systems. I'm especially interested in whether there are additional optimization opportunities I may have overlooked.