AI Model Memory: Why 16 GB of RAM Isn't 16 GB

My program crashed with java.lang.OutOfMemoryError. My machine had 12 GB of RAM free. Both sentences are true, and the reason why is the same reason you cannot run a big AI model on your laptop. Real numbers from my machine below.

The ceiling you never chose

Your program does not use the machine's memory directly. Java creates a separate space for it, the heap, and by default that heap is about a quarter of your RAM. On my 16 GB machine: 4,096 MB. I never chose that.

Why smaller on purpose? Two reasons:

• The garbage collector has to walk everything still alive, and a bigger heap can mean a longer walk (last episode: 6 million living objects cost 29 ms per collection).
• Your program is not the only thing on the machine.

The ceiling is a policy

Here is the part that surprised me. I ran the same allocation loop