Parsers don't have to be complicated

Parsers don’t have to be complicated Table of Contents Introduction Years ago I started writing a proper shader front-end parser for bgfx and used the Lemon parser generator. It was small, it worked, but I never liked the code it produced. The generated output felt hard to read, and the code that generated the parser also felt alien to me. Every time the grammar changed I had to re-learn the shape of the resulting code. It solved the problem, but eventually I abandoned it. On the other side of the spectrum I kept writing ad-hoc parsers for smaller things where using something like Lemon was overkill. Pointer arithmetic, manual loops over characters, a handful of strchr/strncmp style calls, some state variables, and a hope that I hadn’t missed an edge case. These were fast and had no dependencies, but they always ended up repeating the same patterns: skipping whitespace, collecting identifiers, tracking line numbers for error messages, handling the inevitable off-by-one when the input wasn’t perfectly formed. Each new parser became its own little minefield of one-off bugs, and every new thing that needed parsing got its own private copy of them. Most of this ad-hoc parsing code looked roughly like this: 1const char* pos = input.getPtr(); 2const char* end = input.getTerm(); 3 4while (pos "; 19 break; 20 } 21 22 scanner.accept(':'); 23 24 const bx::StringView lineStr = scanner.acceptUntil(")"); 25 26 if (!lineStr.isEmpty() ) 27 { 28 bx::fromString(&line, lineStr); 29 } 30} This is the collapse from the previous section working in your favour. An empty function name is not a valid result, so “missing” and “empty” being indistinguishable is exactly what you want, and the plain return value is the only test needed. Conclusion I first needed this the last time I was writing yet another ad-hoc parser, this time for addr2line output. That produced (now deleted) strConsumeTo , which later became the core of bx::Scanner . By using bx::Scanner I managed to delete: an INI library dependency, and three hand-rolled copies of the same whitespace-and-identifier loop, in URL parsing, path normalization, and stack trace symbolication. If you find yourself writing yet another pointer-chasing loop to skip whitespace and collect an identifier, consider reaching for something like this instead. Parsers really don’t have to be complicated. - One more thing... My mission with bgfx is to empower game developers by providing a cross-platform, graphics API-agnostic rendering library that simplifies porting games across diverse platforms, ensuring seamless performance and compatibility without engine lock-in. If you like this article and support my mission please consider becoming a sponsor! ❤️