Writing terabytes to disk in Go: Stopping the OS Page Cache from eating all your RAM (FADV_DONTNEED)

Hello everyone! This is the second article about the development of RUSEON-core, a Zero-Copy video streaming server for AI platforms and Edge video infrastructure. In the first article, I talked about the fundamental reason why we decided to create our own server in the first place. I also covered the main problem with most similar solutions - the "thundering herd" - and how we managed to squeeze out 8 Gbps on a single CPU core. By the way, I forgot to mention in that article that besides simple streaming, we also record the streams in fMP4 format. It’s stored locally for N amount of time, and it can fly off to an S3 bucket (depending on how long the clients want to keep the recordings). This article is precisely about a non-obvious (well, at least to me, maybe for someone else it's an everyday thing) problem related to data storage and its specifics across all Operating Systems. So, let's dive in. We rolled out our first release to production (100 cameras), made the clients happy, and started working. About an hour passed, and the alerts started flying. I SSH into the server, open htop, and see there's only 100 MB of free RAM. Uh-oh. I should clarify that the production server had 32 gigs of RAM. The expected behavior was that the CPU is chilling, the network card is chewing through the traffic, RAM usage is around 250-300 MB, and the disks are not heavily loaded. So, when you see numbers like that in htop, you start blaming yourself and your crooked hands that wrote this piece of "garbage". But still, we decided to go to Google, ChatGPT, and the like. Fortunately, the answer was found quickly, and we stopped beating ourselves up. The code was absolutely not the culprit; Linux itself ate the memory. If you've ever written tons of data to a disk, I think you already know what’s going on. There is an "invisible enemy" known as the Page Cache. That was exactly the root of this problem. How does the Page Cache work and what to do with it? When your function that is supposed to write data to the disk actually writes data, it doesn't write it to the disk. It writes it to RAM. The logic of the Linux kernel is simple and trivial, and it is aimed at accelerating the "responsiveness" of the system. The whole essence can be explained like this: "Oh, they just wrote a hundred gigabytes of data, they will probably need to read this data soon. Let me keep it in the cache, the user will be happy they could read it so fast." And so it goes, gigabyte after gigabyte, until the server runs out of physical memory. The typical solution to the problem is writing a bash script that runs echo 3 > /proc/sys/vm/drop_caches once an hour. And some people just ignore it and let the system kill random processes via the OOM Killer. But we are building a fault-tolerant thing. That doesn't work for us. The task is to explain to the OS kernel that our fMP4 video archive segments are write-only trash for a short amount of time (because if the client doesn't want to keep records for long, the archive gets cleaned up, and if they do, we send the archive to S3 after N time, and still clean up the local copy). So everything should work on the principle of "write and forget". How to tame the Linux kernel via Go (the right way) In C/C++, there is a system call for this: posix_fadvise. You can explicitly tell the OS exactly how you will be working with the file. In Go, this doesn't exist out of the box. But there is a very good package, golang.org/x/sys/unix, which allows you to easily replace the system call. The flag is called FADV_DONTNEED. We literally say to the kernel: "We wrote it, flush it to disk and get the hell out of the cache." But there is one cruel nuance here, which I stumbled upon myself and racked my brain over for a long time (and all it took was reading the docs, but here, just like with assembling IKEA furniture: "Why do I need a manual, I know how to do it myself"). The Linux kernel does not remove pages from the cache if they are so-called "dirty" - meaning they haven't been physically written to the disk platter yet. If you just call Fadvise, nothing will happen. First, you need to do a hard Sync(). // File: pkg/storage/localfs/file_linux.go package localfs import ( "os" "golang.org/x/sys/unix" ) type FileWrapper struct { *os.File } func (fw *FileWrapper) DropCache() error { // First, flush dirty pages to disk! if err := fw.File.Sync(); err != nil { return err } // And only then order the kernel to forget them return unix.Fadvise(int(fw.File.Fd()), 0, 0, unix.FADV_DONTNEED) } Architecture: How not to break the build on Windows System calls are almost always a huge cross-platform headache. On Windows, the FADV_DONTNEED flag simply does not exist (hello there, Microsoft! Are you guys doing okay?). If you don't split the code for different OSs, the compiler will just tell you to get lost. Therefore, a rather elegant (I invite you to argue this statement in the comments) interface was implemented. In the core of the recorder, there is now a check to see if the file descriptor knows how to drop the cache: // OPTIMIZATION: Saving RAM from Page Cache if dropper, ok := file.(registry.CacheDropper); ok { _ = dropper.DropCache() } And then comes the magic of Go build tags (thank you for those). In the file_linux.go file (with the //go:build linux tag), we call unix.Fadvise. And right next to it lies the file_others.go file (with the //go:build !linux tag), where the DropCache() method just does a regular Sync() and returns. The code is crystal clear, the linters are happy, and the build works everywhere. So, what was the result? We roll out the fixes, launch the exact same 100 cameras. I open the dashboard. The memory consumption graph looks almost like a perfect straight line. The server grabbed its rightful 250 megs for the Go process heap - and that's it. Hooray, victory! There is no more massive Page Cache growth. No processes are being evicted to swap. The server honestly writes tens of gigabytes per hour, and RAM is at peace. By the way, when you are backing up databases, parsing giant logs, or simply writing heavy files - this feature will save you a mountain of headaches. I also don't understand why this flag is barely talked about or written about anywhere. Usually, people only write about how to properly allocate slices, but there is complete silence regarding the fact that at the level of file operations, your OS can reduce all your efforts to zero. In short, in the open-source part of ruseon-core, this logic is now wired deep into the recording engine. The conclusion and advice I want to give is - don't blindly trust the kernel with memory, hoping that the OS is hypothetically "perfect and maximally thought out." Sometimes you have to slap the kernel on the wrist. Otherwise, you will run into similar problems as I did. The source code, as always, is available on GitHub: https://github.com/RUSEGAL/ruseon-core Top comments (0)