Building Distributed Systems in Elixir: Part 5 - Supervisor From Scratch

In the previous part of this series, we explored process links. A linked worker that crashes sends an exit signal to the process linked to it. By default, that failure propagates and can terminate both processes. We also saw that a process can trap exits: Process.flag(:trap_exit, true) Once exit trapping is enabled, an incoming exit signal becomes a mailbox message: {:EXIT, pid, reason} Receiving that message tells us that a linked process terminated, but detection is only the beginning. What should happen next? If the failed process was performing essential work, we may need to start a replacement. In this part, we'll build a small supervisor from scratch using: spawn/1 spawn_link/1 Process.flag(:trap_exit, true) send/2 receive/1 Our supervisor will manage two workers. When one crashes, it will restart only that worker while leaving the healthy sibling unchanged. No GenServer . No OTP Supervisor . The goal is not to replace OTP. The goal is to understand the mechanism and policy that an OTP supervisor provides. The Problem Imagine a long-running worker process: worker = spawn(fn -> worker_loop() end) The worker receives jobs and keeps private state in its recursive loop. If it crashes, the process disappears: Client Worker | | | {:work, value} | |--------------------------->| | | | crash | X | | {:work, another_value} |---------------------------> no process Messages sent to a dead local PID do not bring that process back. The application needs another process responsible for detecting the failure and deciding whether to start a replacement. We could periodically call: Process.alive?(worker) But polling has several problems: - failure detection is delayed until the next check - restart logic becomes scattered through the application - the code must repeatedly check every worker - a liveness check does not tell us why the worker stopped The BEAM already gives us event-driven failure detection through links and exit signals. We need a process that uses those primitives to apply a restart policy. The Design We will create one manual supervisor and two workers: +------------------------+ | Manual Supervisor | |------------------------| | trap_exit = true | | child PID -> name map | | restart loop | +------------------------+ / \ link / \ link v v +------------------+ +------------------+ | worker_1 | | worker_2 | |------------------| |------------------| | receive loop | | receive loop | | private state | | private state | +------------------+ +------------------+ The supervisor will: - enable exit trapping - start and link both workers - record the relationship between each PID and logical worker name - wait for messages - restart a worker after an abnormal exit - leave healthy siblings running - avoid restarting a worker after a normal exit This is a minimal one-for-one restart strategy. The Worker Our worker starts with a logical name and a processed-job count: defmodule Worker do def start(name) do spawn_link(fn -> loop(name, 0) end) end end The call to spawn_link/1 does two things atomically: start process + create link The link connects the worker's failure lifecycle to the process that starts it. In this example, that process is our manual supervisor. The worker then enters a receive loop: defp loop(name, count) do receive do {:work, caller, value} -> new_count = count + 1 send(caller, {:done, name, value, new_count}) loop(name, new_count) :crash -> raise "#{name} crashed" :stop -> IO.puts("#{name} stopping normally") :ok end end The worker supports three messages. Processing Work {:work, caller, value} For each job, the worker increments its private counter and replies: {:done, name, value, new_count} The recursive call carries the new state into the next loop: loop(name, new_count) Crashing Intentionally :crash This branch raises an exception: raise "#{name} crashed" The exception terminates the worker with an abnormal exit reason. Because the worker is linked to the supervisor, an exit signal travels across that link. Stopping Normally :stop Returning :ok lets the worker function finish normally. The worker exits with reason: :normal That distinction will determine whether our supervisor restarts it. Starting the Manual Supervisor The public start/1 function creates the supervisor process: defmodule ManualSupervisor do def start(worker_names) do spawn(fn -> init(worker_names) end) end end We use spawn/1 , rather than spawn_link/1 , here because this example focuses on links between the supervisor and its children. The caller running the demonstration is not part of that supervision relationship. The new supervisor begins in init/1 : defp init(worker_names) do Process.flag(:trap_exit, true) children = Map.new(worker_names, fn name -> pid = start_child(name) {pid, name} end) loop(children) end The order here matters. The supervisor enables exit trapping before it starts linked children: Process.flag(:trap_exit, true) If a child crashes immediately after it starts, the supervisor is already prepared to receive the exit as a message instead of being terminated by it. Tracking Child Identity A restarted process receives a new PID. That means a PID cannot be the permanent identity of a logical worker: worker_1 before crash -> #PID worker_1 after crash -> #PID The name remains :worker_1 , but the process incarnation changes. Our supervisor stores its children in a map: %{ worker_1_pid => :worker_1, worker_2_pid => :worker_2 } The PID is the key because the exit message identifies the terminated process by PID: {:EXIT, pid, reason} With the map, the supervisor can recover the logical name: name = Map.fetch!(children, pid) It can then start a replacement using that name. Starting Children Child creation is kept in one helper: defp start_child(name) do pid = Worker.start(name) IO.puts("supervisor: started #{name} as #{inspect(pid)}") pid end Worker.start/1 uses spawn_link/1 , so the calling supervisor becomes linked to the new worker. After initialization, the relationship looks like this: ManualSupervisor | +---- link ---- worker_1 | +---- link ---- worker_2 The supervisor then carries the child map into its recursive message loop: loop(children) Detecting an Abnormal Exit Suppose the demonstration sends this message: send(worker_1, :crash) The worker raises an exception and terminates. Because the supervisor traps exits, it receives: {:EXIT, worker_1_pid, reason} The abnormal-exit branch handles that message: {:EXIT, pid, reason} -> name = Map.fetch!(children, pid) IO.puts("supervisor: #{name} exited with #{inspect(reason)}") children = Map.delete(children, pid) new_pid = start_child(name) IO.puts("supervisor: restarted #{name} as #{inspect(new_pid)}") loop(Map.put(children, new_pid, name)) There are four important steps here. 1. Identify the Logical Worker name = Map.fetch!(children, pid) The exit message contains the old PID. The child map tells us that this PID belonged to :worker_1 . 2. Remove the Dead PID children = Map.delete(children, pid) The old PID will never become alive again, so it must not remain in the supervisor's state. 3. Start a Replacement new_pid = start_child(name) This creates another linked process with the same logical name and a new PID. 4. Store the Replacement loop(Map.put(children, new_pid, name)) The recursive loop continues with the updated child map. The Restart Message Flow The complete interaction looks like this: Demo Supervisor worker_1 send(worker_1, :crash) -----------------------------------------------> raises exits {:EXIT, pid, reason} {:children, ref, %{worker_1: new_pid, worker_2: same_pid}} worker_1 is replaced worker_2 survives -> worker_2 keeps the same PID That is the meaning of one-for-one supervision in this small example. Healthy Siblings Stay Alive When worker_1 crashes, the supervisor does not rebuild its entire child set. It removes and replaces only the matching PID: children = Map.delete(children, worker_1_pid) children = Map.put(children, new_worker_1_pid, :worker_1) The worker_2 entry is untouched. Before worker_1 -> #PID worker_2 -> #PID After worker_1 crashes worker_1 -> #PID changed worker_2 -> #PID unchanged This matters because a healthy worker may contain useful in-memory state. Restarting it unnecessarily would discard that state and interrupt work it could have continued performing. OTP also provides other restart strategies for cases where child lifecycles are related, but our implementation supports only one-for-one replacement. Normal Exit Is Not a Failure Our supervisor has a separate pattern for a normal exit: {:EXIT, pid, :normal} -> name = Map.fetch!(children, pid) IO.puts("supervisor: #{name} stopped normally; not restarting") loop(Map.delete(children, pid)) When a worker receives :stop , it finishes normally: :stop -> IO.puts("#{name} stopping normally") :ok The supervisor removes that child but does not replace it. This gives our restart policy a simple rule: reason == :normal -> do not restart reason != :normal -> restart This is closest to OTP's :transient restart policy, which restarts a child only after an abnormal exit. It is still deliberately simplified. OTP child specifications provide explicit :permanent , :transient , and :temporary policies instead of forcing every application to encode the decision directly in a receive loop. Asking the Supervisor for Its Children The demonstration needs a safe way to discover current child PIDs, including a replacement PID after a crash. The public function uses a correlated request-reply protocol: def children(supervisor) do ref = make_ref() send(supervisor, {:children, self(), ref}) receive do {:children, ^ref, children} -> children after 1_000 -> {:error, :timeout} end end This reuses an idea from Part 2. The request includes: {:children, caller_pid, reference} The supervisor transforms its internal PID-to-name map into a caller-friendly name-to-PID map: {:children, caller, ref} -> by_name = Map.new(children, fn {pid, name} -> {name, pid} end) send(caller, {:children, ref, by_name}) loop