Tracking down a Zsh history data loss bug

Table of contents For many years, I sometimes discovered that commands I was sure I had run were no longer present in my Z shell history file (~/.zsh_history ). In this article, I will show you how I tracked down the bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy! The good news first Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue - open it after reading this investigation to not spoil the fun. Spoiler: link to the upstream fix The symptom Occasionally, I noticed that commands I knew I executed the day before were not findable in my shell history, meaning pressing Ctrl+R for backward history search yielded no results. Whenever I noticed this, my shell history file contained only very old entries, with years of newer entries missing. The first few times this happened I just restored my shell history from my daily backup and did not bother investigating any further. But the issue kept happening. I noticed that there was no visible corruption in the .zsh_history (no non-printable characters or incomplete lines of text), and that the number of lines in the file was not always the same. What was not clear to me was whether it was Zsh itself, or some other program, or perhaps the combination of multiple zsh(1) processes that caused the issue. My Zsh history config I set the following history-related options in my ~/.zshrc : # Load 4000 lines of history (for Ctrl+R backward search), but save O(∞) HISTSIZE=4000 HISTFILE=~/.zsh_history SAVEHIST=10000000 # Do not save (adjacent) duplicate entries setopt HIST_IGNORE_DUPS # Append history entries to ~/.zsh_history when commands are run. setopt INC_APPEND_HISTORY # …but do not share history (enabled by default in NixOS’s /etc/zshrc). unsetopt SHARE_HISTORY In practice, this means my shells are separate sessions that all stream their commands into a shared ~/.zsh_history . The history is intentionally not shared, so when I want to access entries that another shell wrote, I explicitly run exec zsh . Tracing the behavior When I asked for help on Mastodon in December 2024 (mostly in the hope that somebody else already encountered and diagnosed this problem), one suggestion I got was to use file system change monitoring mechanisms like inotify or fsevents to find the culprit that truncates (or changes?) the Zsh history file. The next sections walk through the available options on Linux which I tried. inotify The inotify(7) Linux kernel subsystem is one of the oldest file system change monitoring APIs available in Linux (released 2005). To get a good understanding of how Zsh modifies the history file, it is not sufficient to monitor just .zsh_history : midna ~ % inotifywait --monitor .zsh_history Setting up watches. Watches established. .zsh_history OPEN .zsh_history ACCESS .zsh_history ACCESS […] .zsh_history ACCESS .zsh_history CLOSE_NOWRITE,CLOSE .zsh_history ATTRIB .zsh_history CLOSE_WRITE,CLOSE .zsh_history DELETE_SELF ^C The file is opened, accessed (= read) and then… deleted?! By monitoring the containing directory, we see the whole picture: midna ~ % inotifywait --monitor ~ /home/michael/ OPEN .zsh_history /home/michael/ ACCESS .zsh_history /home/michael/ ACCESS .zsh_history […] /home/michael/ ACCESS .zsh_history /home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history /home/michael/ CLOSE_WRITE,CLOSE .zsh_history /home/michael/ OPEN .zsh_history /home/michael/ CLOSE_WRITE,CLOSE .zsh_history /home/michael/ OPEN .zsh_history /home/michael/ ACCESS .zsh_history /home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history /home/michael/ CREATE .zsh_history.new /home/michael/ OPEN .zsh_history.new /home/michael/ ATTRIB .zsh_history.new /home/michael/ MODIFY .zsh_history.new /home/michael/ CLOSE_WRITE,CLOSE .zsh_history.new /home/michael/ MOVED_FROM .zsh_history.new /home/michael/ MOVED_TO .zsh_history /home/michael/ CLOSE_WRITE,CLOSE .zsh_history So Zsh reads the old history file contents, writes them to a new file, then renames the new file over the old one, thereby deleting the old one. Now it makes sense! Unfortunately, we do not see the process IDs (PIDs) of the responsible process for the file system event, not even with the sibling utility fsnotifywait(1) , which uses fanotify(7) , an API that does provide this information! I checked, and the kernel does send the PID, but fsnotifywait does not display the PID. fatrace Luckily, there is fatrace(8) , which does display the process name and PID. Here is what Zsh’s history rewriting looks like with fatrace(8) : zsh(197994): CWO /home/michael/.zsh_history zsh(197994): O /home/michael/.zsh_history zsh(197994): R /home/michael/.zsh_history zsh(197994): R /home/michael/.zsh_history […] zsh(197994): R /home/michael/.zsh_history zsh(197994): C /home/michael/.zsh_history zsh(197994): + /home/michael zsh(197994): O /home/michael/.zsh_history.new zsh(197994): W /home/michael/.zsh_history.new zsh(197994): W /home/michael/.zsh_history.new zsh(197994): W /home/michael/.zsh_history.new […] zsh(197994): W /home/michael/.zsh_history.new zsh(197994): CW /home/michael/.zsh_history.new zsh(197994): <> /home/michael zsh(197994): CW (deleted) zsh(197994): C /nix/store/80vwnjjgcrbp41pk927r8lzybjhy0k73-zsh-5.9.1/bin/zsh […] This gives us the PID, so now we can verify whether multiple processes were involved in corrupting the shell history. But, we don’t have any insight into how much data each Zsh PID is reading/writing, so even with a fatrace log, it would still not be clear what happened. strace Of course, one could use strace(1) , in particular with its -k flag, to further look into Zsh behavior, but it seems like a logistical nightmare to arrange for every (interactive) Zsh process to get a corresponding strace run, and I was not sure if always-stracing a shell changes behavior in subtle ways, so I did not pursue the strace route. (Once I had a reproducer, strace became easy enough to use and very helpful.) bpftrace To get more visibility into Zsh’s read and write operations, we can reach for bpftrace(8) . To get started, I created the following bpftrace program, which is run on every open(2) syscall and logs which process opened the .zsh_history file, including the user stack trace: tracepoint:syscalls:sys_enter_open, tracepoint:syscalls:sys_enter_openat, tracepoint:syscalls:sys_enter_openat2 /str(args.filename) == "/home/michael/.zsh_history" || str(args.filename) == ".zsh_history"/ { printf("%-6d %-16s open(%s)%s", pid, comm, str(args.filename), ustack); } On NixOS 26.05, I can run the program as follows: midna ~ % nix shell nixpkgs#bpftrace midna ~ 2 % sudo bpftrace path.bt Attached 3 probes 212030 zsh open(/home/michael/.zsh_history) __internal_syscall_cancel+142 __syscall_cancel+20 __libc_open64+87 lockhistfile+642 readhistfile+2213 zsh_main+1118 __libc_start_call_main+117 __libc_start_main_alias_2+136 _start+37 212030 zsh open(/home/michael/.zsh_history) __internal_syscall_cancel+142 __syscall_cancel+20 __libc_open64+87 _IO_file_open+51 _IO_file_fopen@@GLIBC_2.2.5+303 __fopen_internal+134 readhistfile+2277 zsh_main+1118 __libc_start_call_main+117 __libc_start_main_alias_2+136 _start+37 212030 zsh open(/home/michael/.zsh_history) __internal_syscall_cancel+142 __syscall_cancel+20 __libc_open64+87 lockhistfile+642 savehistfile+165 zexit+204 zsh_main+1522 __libc_start_call_main+117 __libc_start_main_alias_2+136 _start+37 212030 zsh open(/home/michael/.zsh_history) __internal_syscall_cancel+142 __syscall_cancel+20 __libc_open64+87 savehistfile+752 zexit+204 zsh_main+1522 __libc_start_call_main+117 __libc_start_main_alias_2+136 _start+37 212030 zsh open(/home/michael/.zsh_history) __internal_syscall_cancel+142 __syscall_cancel+20 __libc_open64+87 _IO_file_open+51 _IO_file_fopen@@GLIBC_2.2.5+303 __fopen_internal+134 readhistfile+2277 savehistfile+2498 zexit+204 zsh_main+1522 __libc_start_call_main+117 __libc_start_main_alias_2+136 _start+37 ^C Encouraged by this early success, I extended the program as follows to cover more system calls: Full zshhisttrace.bt bpftrace code #!/usr/bin/bpftrace #include #include tracepoint:syscalls:sys_enter_open /comm == "zsh"/ { printf("%s(%d) open: %s flags %x mode %x\n", comm, pid, str(args->filename), args->flags, args->mode); } tracepoint:syscalls:sys_enter_openat { if (!strcontains(str(args->filename), "zsh_history")) { delete(@openfn[tid]); return; } @openfn[tid] = 1; printf("%s(%d) openat: ", comm, pid); if (args->dfd dfd); } printf("%s flags %x mode %x\n", str(args->filename), args->flags, args->mode); } tracepoint:syscalls:sys_exit_openat /@openfn[tid]/ { @reads[tid,(int64)args->ret] = 1; // TODO: bpftrace 0.22 introduces has_key @writes[tid,(int64)args->ret] = 1; // TODO: bpftrace 0.22 introduces has_key } tracepoint:syscalls:sys_enter_close /@reads[tid,(int64)args->fd]/ { printf("%s(%d) close %d (reads: %d, writes: %d)\n", comm, pid, args->fd, @reads[tid,(int64)args->fd]-1, @writes[tid,(int64)args->fd]-1); delete(@reads[tid,(int64)args->fd]); delete(@writes[tid,(int64)args->fd]); } // tracepoint:syscalls:sys_enter_openat2 /comm == "zsh"/ { // printf("%s(%d) openat: ", comm, pid); // if (args->dfd dfd); // } // printf("%s \n", str(args->filename)); // } tracepoint:syscalls:sys_enter_rename /comm == "zsh"/ { printf("%s(%d) rename:", comm, pid); printf("%s -> %s\n", str(args->oldname), str(args->newname)); } tracepoint:syscalls:sys_enter_symlink /comm == "zsh"/ { printf("%s(%d) symlink ", comm, pid); printf("%s -> %s\n", str(args->oldname), str(args->newname)); } tracepoint:syscalls:sys_enter_unlink /comm == "zsh"/ { printf("%s(%d) unlink ", comm, pid); printf("%s\n", str(args->pathname)); } tracepoint:syscalls:sys_enter_unlinkat /comm == "zsh"/ { printf("%s(%d) unlinkat ", comm, pid); printf("%s\n", str(args->pathname)); } tracepoint:syscalls:sys_enter_lseek /comm == "zsh"/ { printf("%s(%d) lseek fd %d offset %d whence %d\n", comm, pid, args->fd, args->offset, args->whence); } tracepoint:syscalls:sys_enter_read /@reads[tid,(