Practical notes on setting up a GPU kernel-dev environment: CUDA 13's GPU cutoff, async timing pitfalls, profiler counter permissions [P]

I'm working through a free series on GPU kernel engineering (my own channel; I posted the roadmap here earlier: https://www.reddit.com/r/MachineLearning/comments/1xa4h2g/open_gpu_kernel_engineering_series_naive_gemm_to/ ). Session 1 is a setup session. While preparing it I collected a few things that cost me time and that I rarely see in one place, so I'm sharing them here directly. The video and code are linked at the end.

Check compute capability, not just "has an NVIDIA GPU"

nvidia-smi --query-gpu=name,compute_cap --format=csv

7.5 is Turing, 8.0/8.6 Ampere, 8.9 Ada, 9.0 Hopper. This decides what you can do:

• CUDA 13 dropped Maxwell, Pascal and Volta. For those cards, install a CUDA 12 toolkit.
• GTX 16-series is Turing but has no tensor cores.
• Tensor core work needs RTX 20-series or newer (or a T4); Ampere-specific instructions need Ampere or newer.
• FP8 needs Ada or Hopper. Hopper-specific features (TMA, WGMMA) need an H100, which most people will have to rent.

Driver, toolkit and PyTorch's runtime are three different things

The CUDA version in the nvidia-smi header is the newest version your driver supports, not what's installed. pip-installed PyTorch bundles its own CUDA runtime, so torch.cuda.is_available() returning True says nothing about whether nvcc exists.

On WSL 2, don't install a Linux driver inside the distro; use the WSL-Ubuntu toolkit-only installer, because the Windows driver is used.

GPU launches are asynchronous, so naive timing measures queueing

A Python timer around x @ y mostly times the launch, not the work. Warm up, synchronize, use CUDA events, average over several runs:

def bench(fn, iters=20):
    for _ in range(3):
        fn()
    torch.cuda.synchronize()
    s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    s.record()
    for _ in range(iters):
        fn()
    e.record()
    torch.cuda.synchronize()
    return s.elapsed_time(e) / iters # ms

For a matmul, throughput is about 2*M*N*K FLOPs divided by time. Record the dtype next to the number, since FP32 and BF16 baselines differ a lot.

Profiler counters: ERR_NVGPUCTRPERM

By default, non-admin users can't read GPU performance counters, so Nsight Compute fails with this error. Fixes: run ncu with sudo (use the full path, since sudo usually drops your PATH), or have an admin enable counter access for all users. On Windows/WSL it's enabled from the Windows side.

If you rent GPUs, test this before committing to a provider, because managed container platforms can block counter access.

Smoke test before anything else

A tiny add-one kernel with an error check on every CUDA call, built with nvcc -arch=native, then nsys and ncu run on it once. If that works, the driver, toolkit, GPU and both profilers are all fine.

The session also covers the prerequisites (pointers, flat matrix indexing, matmul and its FLOP count, PyTorch basics) and compares ways to get a GPU. It runs about 40 to 45 minutes.

Video:

Code: https://github.com/prasoonmhwr/gpu_kernel_engineering (folder: session-01-prerequisites-setup)

Caveats: the commands are written for Ubuntu and I haven't tested every distro.

Questions for people who do this daily:

• What setup trap cost you the most time? I'm sure I've missed some (Secure Boot and driver loading, container toolkits, mismatched host compilers).
• Is anything above wrong or out of date for your setup?