How async processing hides latency and improves responsiveness

Asynchronous vs. Synchronous Processing

Asynchronous processing enables tasks to execute independently and in overlapping periods, unlike synchronous processing, where tasks run in sequence. In other words, in a synchronous system, a task is executed to completion before the next one can begin.

For example, suppose a single-threaded synchronous server is reading and processing messages. It first reads a message from a socket, which may require the thread to block until the message fully arrives. The server then processes the message, and it finally sends a response before starting to process the next message. If request processing takes a long time or blocks, the system waits synchronously.

Asynchronous processing removes this constraint, allowing multiple tasks to progress simultaneously. For example, a server using asynchronous processing can process multiple independent requests concurrently by using I/O multiplexing, an operating system (OS) interface, to poll for the status of various connections. The server can then react to events, such as the socket becoming readable or writeable, to process a request. Similarly, the asynchronous server can initiate sending a response over the network and then work on other tasks without waiting for the response.

Asynchronous processing is similar to concurrent programming. However, asynchronous processing differs from concurrent programming because it has explicit interfaces. For example, concurrent programming using threads allows a server to synchronously process a request while retaining concurrency by context-switching between threads. The server executes the send() and recv() system calls, which block if there's nothing to read from a socket or the socket is not writeable. When the server blocks, the OS switches to another thread for concurrent execution.

In contrast, with asynchronous processing, the server uses an I/O multiplexing interface to poll for socket state. The I/O multiplexing interface tells the server what sockets are readable, and the server can read from them without blocking the thread. Similarly, when the server sends a response, it uses an asynchronous interface to send the response, but it can then immediately continue work without blocking, letting the OS send the response in the background.

Figure 1 illustrates the difference between synchronous and asynchronous processing (which are also summarized in the sidebar titled "Differences between asynchronous, concurrent, and parallel processing"). In this example, we have two tasks, A and B, that must run to finish our work in full. Suppose a backend system needs to communicate with external systems A and B to complete a request it has received.

In synchronous processing, each task must finish before the next one starts. We run task A until it is complete, including the I/O it submits, and then run task B. The total time needed is the time of all tasks added together. If a backend service needs to do all these tasks, users must wait for this total time to get their response.

However, in asynchronous processing, we can perform the I/O for task A simultaneously with task B, and we are finished when both of them are done. If the I/O runs simultaneously, the wait time is much shorter for users, even though each task still takes the same time. This works well in backend services when making database calls or calling other services because these tasks can run independently without blocking each other. But there's a catch: if your I/O can't run in parallel, using asynchronous processing won't help but could instead make things run slower because managing async tasks adds extra work for the system.

Differences between asynchronous, concurrent, and parallel processing

Concurrent processing means executing tasks at the same time through multiplexing on the same compute unit, and parallel processing means executing tasks simultaneously on different compute units. While this distinction may seem subtle, the takeaway is that concurrent processing is about structuring applications in a way conducive to executing multiple things despite them potentially running sequentially on the same compute unit. Parallel processing, on the other hand, is about performing various things on different compute units, reducing execution time.

Although asynchronous processing is related to concurrent and parallel processing, it is fundamentally about structuring your code to handle tasks that might take time to complete. In other words, asynchronous processing can enable both concurrency and parallelism, but it doesn't guarantee either. For example, you might write asynchronous code that runs concurrently on a single CPU core by switching between tasks, or you might have asynchronous code that runs in parallel across multiple cores.

Asynchronous processing is also a critical technique for hiding latency. Some operations take a long time to complete, despite your best efforts to reduce latency, so it is essential to perform operations without everyone having to wait for them to complete. For example, backend systems typically interact with external systems like third-party services, database servers, and message queues, where each interaction adds some latency. With synchronous processing, you often build systems that don't exploit the inherent parallelism available and that cause idle time where you're waiting for systems to complete their work. In contrast, async processing allows you to minimize wait time by starting operations asynchronously and reacting when they are complete.

In synchronous processing, you structure your code as a sequence of operations that depend on each other. For example, a request processing function for a synchronous server might look something like the following.

Listing 1: A simple example of a synchronous system

fn process_requests(socket: &Socket) {
    loop {
        process_request(socket);
    }
}

fn process_request(socket: &Socket) {
    let msg = socket.recv();
    let request = parse_message(msg);
    let resp = match request {
        Request::GetUserInfo(id) => get_user_info(id);
    };
    let resp = format_response(resp);
    socket.send(resp);
}

At a high-level, we have the process_requests function, which processes any incoming requests from a socket. In the process_request function, each step is run to completion before we start another step. We read a message from the socket, we parse the message to determine what the request is, we process the request, and we finally send a response over the socket. More importantly, we don't start another process_request until we've sent out a response, and we don't allow requests to be processed from multiple sockets either.

While concurrency primitives like coroutines and futures enable parallel execution, they're insufficient for efficient asynchronous processing, particularly for I/O. You must structure the application differently if a server processes thousands of concurrent connections. The event loop is the foundation for efficiently multiplexing I/O operations across many connections.

The Event Loop

The event loop is the central coordinator for all input and output operations-it's at the heart of an asynchronous system. While traditional synchronous programs handle one connection at a time-like a single worker processing tasks in sequence-an event loop operates as a dispatcher, simultaneously managing thousands of I/O operations. This architectural pattern, sometimes called an I/O loop or I/O dispatcher, is how asynchronous processing handles concurrent operations efficiently. Instead of dedicating separate resources to each connection, the event loop multiplexes various I/O sources-network connections, file operations, timers, and more-by tracking their states and processing them when they're ready.

"The event loop is the central coordinator for all input and output operations-it's at the heart of an asynchronous system."

The event loop follows a simple yet powerful pattern:

• Poll for events.
• Process events.
• Run scheduled tasks.
• Repeat.

The event loop polls for events such as incoming data from a socket, an expired timer, or I/O completion by using OS-specific I/O multiplexing interfaces such as io_uring and epoll on Linux, kqueue on macOS, and IOCP on Windows. These interfaces let you register interest in an event source and get a notification when an event happens. For example, instead of reading data from a socket, the application expresses interest in a socket becoming readable. When data arrives from the network to the socket, the OS notifies the application, via the I/O multiplexing interface, that the socket is now readable. The event loop discovers this via polling and calls into the application's event handling logic to process the newly arrived data from the socket.

Let's implement a basic event loop in Rust to understand its structure better:

struct EventLoop {
    // Holds registered event sources like sockets, files, timers
    sources: Vec<Source>,
}

impl EventLoop {
    fn run(&mut self) {
        loop {
            // Create a new collection to store events
            let mut events = Events::new();

            // Poll for new events with a timeout
            self.poll(&mut events, Duration::from_millis(100));

            // Process each event that was found
            for event in events.iter() {
                self.process_event(&event);
            }

            // Run any scheduled tasks
            self.run_scheduled_tasks();
        }
    }
}

The EventLoop::run() method demonstrates the core functionality of event-driven programming: continuously polling for and processing events. The poll() method uses an OS-specific I/O multiplexing interface, such as io_uring, for events on event sources. As you can see in the example code, we also specify a timeout for event polling. A timeout is needed because I/O polling in the event loop is often the only synchronous code that blocks the thread until an event happens. Polling can block if the system is idle and no events occur, and this blocking can reduce the wasted CPU cycles when there's nothing to do. However, to ensure that the event loop does not block forever, the timeout ensures that we return from poll(). This allows the event loop to also perform work that is not conditional to an event, such as executing background work. However, in some cases, you might use busy-polling to avoid the sleep/wakeup cycle latency for some latency-sensitive event loops.

The process_event function is responsible for processing any events discovered during polling. For example, if the application registered interest in data arriving from the network (such as a socket becoming readable), the process_event function reads from the socket and forwards the data for the application to process. A simple process_event function might look something like this:

struct EventLoop {
    // Holds registered event sources like sockets, files, timers
    sources: Vec<Source>,
}

impl EventLoop {
    fn run(&mut self) {
        loop {
            // Create a new collection to store events
            let mut events = Events::new();

            // Poll for new events with a timeout
            self.poll(&mut events, Duration::from_millis(100));

            // Process each event that was found
            for event in events.iter() {
                self.process_event(&event);
            }

            // Run any scheduled tasks
            self.run_scheduled_tasks();
        }
    }
}

As you can see, each event is represented by an Event enumeration with variants for different events. The event-processing logic is specific to how the event loop is structured. For example, if the event loop uses callbacks for event handling, it calls them, delegating work to the application. The application may then perform the work in the callback or submit the work to another thread for processing.

Figure 10.2 visualizes how the event loop performs work. In this example, work is split into three separate tasks:

• Accept connection
• Process request
• Send response

The first task runs when the I/O multiplexer notifies the event loop that there is an incoming connection. The application reacts to the event by accepting the connection and then registering interest about when the accepted socket becomes readable. When data arrives from the network, the OS notifies the event loop that the socket is readable. The application reacts to this by reading from the socket and processing the incoming request. Finally, the application registers interest in the socket becoming writable. When the OS has enough buffer memory for an