I Built a Deep Learning Framework from Scratch in C++ and CUDA (And Beat PyTorch's Speed Multiple Run)

If you work in AI today, you are almost certainly using PyTorch or TensorFlow. They are incredible tools, but they are also massive black boxes. As an AI developer, I realized that relying solely on these frameworks meant I didn't truly understand the underlying hardware realities, memory mechanics, or how the math actually maps to GPU acceleration.

So, I decided to strip away the abstractions. Over the last few months, I built Aakaar - a deep learning framework developed completely from scratch using native C++ and CUDA, wrapped in Python for ease of use. Here is how I built it, the engineering challenges I faced, and how it surprisingly edged out PyTorch in a direct benchmark.

The Architecture: Hand-Coding the Math

Building a framework from scratch means you don't get autograd for free. The core architecture of Aakaar relies on:

• Backend: A purely native C++ implementation.
• Components: 18 hand-coded native loss modules and 11 custom optimizers built directly into the C++ backend.
• Memory Management: Explicit contiguity management during tensor transpositions and custom backpropagation. Instead of relying on an automated computational graph, I had to manually track and allocate memory layouts during the backward passes.
• Frontend: A Python wrapper so models can be defined intuitively, similar to standard frameworks.

The Hardest Part: Memory on the GPU

The primary engineering hurdle wasn't the forward pass-it was the backward pass. Mapping abstract mathematical shapes directly to physical GPU memory layouts during backpropagation is brutally unforgiving. If your memory isn't strictly contiguous when a tensor transposes during a CUDA kernel execution, the entire system bottleneck slows to a crawl or crashes. Managing that contiguity explicitly in C++ without standard autograd overhead was a massive exercise in systems engineering.

The EMNIST Benchmark: Aakaar vs. PyTorch

To see if Aakaar was structurally viable, I didn't want to just run a toy matrix multiplication. I set up a 5-epoch training loop on the EMNIST dataset and benchmarked Aakaar directly against PyTorch on my local system (Intel i7, RTX 4060, 8GB VRAM). To ensure a fair test, the model architecture, hyperparameters, and weights were identical across both frameworks.

The Results (Average of multiple test runs):

• Aakaar: 127.76 seconds (83.30% acc)
• PyTorch: 131.23 seconds (82.55% acc)

Why was it faster? Aakaar consistently maintained absolute convergence parity while delivering a slight edge in runtime speed. This performance advantage stems directly from bypassing the Python runtime overhead during the low-level C++ optimizer steps. By keeping the optimizer operations strictly in the native backend, Aakaar shaved off the milliseconds of overhead that accumulate over thousands of batches.

What's Next and How to Contribute

Seeing the math align perfectly with the hardware performance made the struggle worth every line of code. Aakaar is fully open-source. I am currently looking for feedback from the systems engineering and AI infrastructure communities. Specifically, if you have experience with CUDA kernel optimization or C++ memory allocation strategies, I would love for you to audit the architecture.

• Test Run (Source Code): https://github.com/aaravaggarwal3535/aakaar-wheels/blob/main/main.ipynb
• Documentation: https://aakaar.readthedocs.io

Have you ever tried building ML components from scratch? Let me know your thoughts or optimization ideas in the comments below!