What I Learned After Building a Programming Language From Scratch in Python

I didn't just write a programming language. I built the pipeline that makes the language work. 🔗 NexPro on GitHub: https://github.com/probal2005/NexPro There are thousands of programming languages in existence. So building another one sounds unnecessary. But my goal with NexPro was never to compete with Python, JavaScript, Rust, or C++. I wanted to answer a much simpler question: What actually happens between writing code and getting an output? So instead of only learning the theory of lexers, parsers, ASTs, interpreters, and runtimes, I decided to implement them. This post documents what I have actually built, what works today, what I learned, and what still needs to be done. 1. The Experiment NexPro is an experimental programming language implemented in Python. Its source files use the: .pa extension. A simple NexPro program looks like this: name = "Probal" city = "Kolkata" say name say city The important part isn't that this syntax is simple. The important part is what happens internally. The source doesn't go directly from: .pa file to: output Instead, it passes through multiple stages. 2. The Actual Language Pipeline The core idea behind NexPro is: ┌───────────────────┐ │ NexPro Source │ │ .pa │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Lexer │ │ Source → Tokens │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Parser │ │ Tokens → AST │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ AST │ │ Program Structure │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Interpreter │ │ Execute Nodes │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Runtime │ │ Values & State │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Output │ └───────────────────┘ This pipeline isn't just a diagram for documentation. These are the actual conceptual components implemented in the repository. 3. Evidence #1 - The Repository Has a Language Implementation Structure The project is organized around the language itself: NexPro/ │ ├── nexpro/ │ ├── cli.py │ ├── lexer.py │ ├── parser.py │ ├── interpreter.py │ ├── runtime.py │ ├── tokens.py │ ├── ast.py │ ├── errors.py │ └── version.py │ ├── examples/ │ ├── hello.pa │ └── variables.pa │ ├── tests/ │ ├── README.md ├── LICENSE └── pyproject.toml Each part exists for a reason. | Component | Responsibility | |---|---| lexer.py | Converts source text into tokens | tokens.py | Defines token types | parser.py | Builds program structure | ast.py | Represents syntax as nodes | interpreter.py | Executes the AST | runtime.py | Handles runtime behavior/state | errors.py | Language-level error handling | cli.py | Provides the command-line interface | tests/ | Tests language behavior | This separation is important because language implementations become difficult to maintain when lexing, parsing, execution, and runtime logic are mixed together. 4. Evidence #2 - NexPro Actually Executes .pa Programs A language project shouldn't stop at syntax diagrams. It needs to execute programs. NexPro provides a CLI command: nexpro run examples/hello.pa For a program such as: say "Hello NexPro!" the interpreter produces: Hello NexPro! Variables can also be used: name = "Probal" city = "Kolkata" say name say city which produces: Probal Kolkata That gives us a complete path: hello.pa ↓ CLI ↓ Lexer ↓ Parser ↓ AST ↓ Interpreter ↓ Output 5. Evidence #3 - Tokens Exist Before Execution Consider: name = 10 A programming language implementation doesn't need to treat this as one giant string. The lexer can break it into meaningful pieces: IDENTIFIER(name) ASSIGN(=) NUMBER(10) That transformation is fundamental. The lexer answers: "What are the pieces?" The parser answers: "How are those pieces related?" The interpreter answers: "What should those relationships do?" This separation is one of the biggest things I understood while building NexPro. 6. Evidence #4 - The AST Changes Everything Consider: a = 10 + 20 The parser doesn't simply need to remember: "10 + 20" It can represent the expression structurally: Assignment / \ a Binary(+) / \ 10 20 This is the Abstract Syntax Tree. The AST gives the interpreter a structured representation of what the programmer wrote. Instead of asking: "What characters are in this string?" the interpreter can ask: "What kind of node am I executing?" That distinction is fundamental to language implementation. 7. Evidence #5 - Arithmetic Becomes a Tree For: a = 10 b = 20 say a + b the important expression is: a + b Conceptually: + / \ a b The interpreter can then resolve: a → 10 b → 20 and evaluate: 10 + 20 giving: 30 This is where a simple syntax feature starts demonstrating the full language pipeline. 8. What Building the Lexer Taught Me At first, tokenization looks easy. You see: a = 10 and think: Split the string. But real language syntax quickly introduces problems. What happens with: a = 10 + 20 What about: name = "Probal" What about: say "Hello World" Now the lexer needs to distinguish: IDENTIFIER NUMBER STRING ASSIGN PLUS SAY It also needs to deal with things such as: Whitespace Unknown characters Strings Numbers Identifiers Operators End of file That was one of my first major lessons: A programming language begins before parsing. 9. What Building the Parser Taught Me The parser is where syntax becomes structure. For example: a = 10 + 20 is not equivalent to: a + 10 = 20 The parser needs to understand relationships and precedence. Even simple arithmetic starts raising questions: 10 + 20 * 5 Should it mean: (10 + 20) * 5 or: 10 + (20 * 5) Language design quickly turns into a combination of: Syntax + Grammar + Precedence + AST design That's something I didn't fully appreciate before implementing it. 10. What Building the Interpreter Taught Me The interpreter is where the language becomes executable. It has to understand things such as: Number String Variable Assignment Binary Expression Say For example: x = 100 say x requires the runtime to maintain state: Environment x → 100 Then: say x requires the interpreter to retrieve: x → 100 and send the value to the output. This is where concepts like scope, environments, values, evaluation, and runtime state become practical instead of theoretical. 11. Why I Chose Python NexPro is currently implemented in Python. That was intentional. For an experimental interpreter, Python gives me: - Fast iteration - Easy testing - Simple data structures - Classes for AST nodes - Dictionaries for environments - Straightforward exception handling I can focus on: language design + lexing + parsing + AST + interpretation without initially having to solve low-level implementation problems. That doesn't mean Python is necessarily the final implementation language. It means Python is a practical place to start. 12. Testing Is Not Optional A language implementation can break very easily. For example: Lexer change ↓ Token change ↓ Parser breaks ↓ AST changes ↓ Interpreter breaks That's why NexPro includes tests. The project has a dedicated: tests/ directory for language behavior. As NexPro grows, I want the test suite to cover: Lexer ↓ Parser ↓ AST ↓ Interpreter ↓ Runtime ↓ CLI The goal is to make every new language feature measurable and reproducible. 13. What NexPro Can Do Today The current implementation is intentionally small. The project has already moved beyond a source-code mockup into an executable interpreter with components for: ✓ CLI execution ✓ Lexical analysis ✓ Tokens ✓ Parsing ✓ AST representation ✓ Variables ✓ Assignment ✓ Strings ✓ Numbers ✓ Basic expressions ✓ Interpretation ✓ Runtime structure ✓ Error handling structure ✓ Tests The exact supported syntax will continue changing as the language develops. That distinction matters. I'm documenting what exists rather than presenting planned features as completed features. 14. What NexPro Cannot Do Yet This is equally important. NexPro is not yet: ❌ A production compiler ❌ A Python replacement ❌ A high-performance language ❌ A mature standard library ❌ A complete ecosystem ❌ A stable 1.0 language There are still major areas to develop: Functions Loops Conditionals Collections Modules Type system Standard library REPL Tooling Debugger Package management Performance And that's exactly why I consider it an interesting engineering project. 15. Roadmap My roadmap is currently divided into three stages. Stage 1 - Language Core ✓ Lexer ✓ Tokens ✓ Parser ✓ AST ✓ Interpreter ✓ Variables ✓ Basic expressions ✓ CLI → Conditionals → Loops → Functions → Collections Stage 2 - Developer Experience → REPL → Better diagnostics → Formatter → Documentation → VS Code syntax highlighting → Debugging tools → Improved testing Stage 3 - Advanced Runtime → Modules → Standard library → Package system → Bytecode → Performance improvements → Possible compilation These are goals, not completed features. 16. Why This Project Matters to Me The biggest result of NexPro isn't the syntax. It's the understanding. Before building a language, I knew the words: Lexer Parser AST Interpreter Runtime After implementing them, I started seeing programming languages as pipelines. When I write: say 10 + 20 I can now mentally see: SOURCE ↓ TOKENS ↓ SYNTAX ↓ AST ↓ EVALUATION ↓ RUNTIME ↓ 30 That shift in understanding is the real reason I started NexPro. 17. What I Want to Investigate Next The next interesting question isn't just: "What syntax should NexPro have?" It's: "How far can I take a language that started as a Python interpreter?" Some questions I want to explore: Can NexPro have a real type system? For example: age: number = 20 name: string = "Probal" Can it compile to bytecode? Instead of: Source → AST → Interpreter potentially: Source ↓ AST ↓ Bytecode ↓ Virtual Machine Can the language have its own package system? Something like: import math Can it eventually have an IDE experience? For example: NexPro ├── Language Server ├── Formatter ├── Debugger └── VS Code Extension Those are future experiments. 18. The Repository Is the Evidence Rather than asking readers to trust a description, I want the repository to be the evidenc