A Remote Function Call Will Never Really Be Local: Rethinking Distributed Computing #4

The API can hide the distance. It cannot erase what the distance changes. "Explore the CAP theorem through practical examples of replication, network partitions, split brain, quorum, consistency, and availability and learn what distributed systems sacrifice when communication fails." Suppose a client needs some work done by another component. On one machine, this is boring in the best possible way. Call a procedure. Pass some arguments. The procedure runs. Get the result back. ✨Continue✨ Then we move that procedure onto another computer. The first instinct is almost irresistible: Why should the application care? If the remote machine exposes the same operation, perhaps we can make calling it look exactly like calling a local procedure. Instead of forcing every application developer to think in sockets, packets, addresses, message formats, and transport protocols, give them something familiar. result = calculate(x, y) The caller should not have to stare into the abyss and think: - open socket - encode x - encode y - construct message - send bytes - wait - receive bytes - decode result That is the promise of Remote Procedure Call, or RPC. Call a procedure on another machine just as you would call a local procedure. Birrell and Nelson’s original RPC design was built around exactly this idea: Provide communication between programs across a network through the familiar abstraction of a procedure call. (Microsoft) It is an excellent abstraction. Which is why it is worth understanding where the abstraction stops. The Network Does Not Know What a Function Is Before RPC can hide the network, something still has to use the network. Distributed communication commonly happens over IP using transports such as TCP or UDP. TCP gives us a connection-oriented byte stream. UDP gives us connectionless datagrams. Neither one understands: calculate(x, y) The network understands messages and bytes. So somewhere between the line of application code and the remote implementation, the procedure call has to be transformed into communication. That is where RPC begins doing its little stage trick. The application calls a client stub as though it were an ordinary procedure. The stub builds a message and passes it to the communication middleware. The message crosses the network. On the server, middleware passes it to a server stub, or skeleton. The server side reconstructs the arguments and calls the real local implementation. The implementation returns a result. Then the whole journey happens in reverse. Server implementation → skeleton → middleware → network → client middleware → client stub → application. The application sees: result = calculate(x, y) The distributed system sees a round trip between two independent processes. The ONC RPC specification describes essentially this model: the caller sends a call message containing the procedure parameters, waits for a reply message containing the results, and resumes once that reply arrives. (RFC Editor) The procedure call did not cross the network. A representation of the procedure call did. That distinction is going to become expensive. Even the Arguments Cannot Travel as Themselves Suppose x is an integer. Easy enough. Put the integer into the message. Except two machines may not even represent that integer identically. One architecture may use one byte order. Another may use another. Strings may have different encoding or alignment rules. Floating-point representations have to agree. Arrays and structures have to be flattened into something that can travel and reconstructed at the other end. Memory addresses are even more suspicious. A pointer that means something inside the client’s address space does not suddenly become meaningful inside the server’s address space. So the stub has another responsibility. Marshalling. Data is encoded into a transmissible representation. On the receiving side it is unmarshalled, or decoded, back into usable data structures. Protocols therefore need agreed representations for parameters and results. ONC RPC, for example, uses External Data Representation to describe the messages exchanged between participants. (RFC Editor) This is the first crack in the illusion. A local call can hand another procedure values that already live in the same computational world. A remote call has to translate one machine’s world into something another machine can reconstruct. The API may look local. The data most definitely knows it travelled. Then We Have to Find the Procedure There is another detail a local call gets almost for free. When code calls a local procedure, the runtime knows where that procedure is. A remote service may be somewhere else entirely. Which machine? Which process? Which endpoint? Which version? So remote invocation needs binding. A client has to discover and connect to the service that implements the desired interface. That may involve some kind of directory or registry. Once bound, the client can continue pretending that it simply possesses something callable. This is location transparency doing useful work. The client wants: Give me the service that performs this operation. It does not necessarily want: Please make me personally manage the physical location and network configuration of the process currently implementing it. Even formal RPC specifications separate the call protocol from the higher-level mechanism that binds a client to a particular service and transport endpoint. (RFC Editor) Again, perfectly reasonable abstraction. Again, the machinery underneath has not disappeared. We have hidden where the call goes. We still have to deal with what happens while it is going there. Local Calls Have a Very Comfortable Timeline Consider an ordinary synchronous procedure call. You call. You wait. The procedure returns. You continue. RPC can reproduce that programming model. But once the operation becomes communication, other interaction patterns suddenly become useful. The client may send a message and wait for a response. Or send without waiting at all: fire and forget. Or send and wait only for an acknowledgement that the request was received. A client may block. It may poll. It may provide a callback. The call may be asynchronous or use deferred synchronization. The ONC RPC model itself notes that implementations are not restricted to the simple blocking model; asynchronous execution is possible so the client can continue doing useful work while the remote operation proceeds. (RFC Editor) Why does a supposedly local-looking call suddenly need all these options? Because latency exists. A local function call is usually close enough that waiting is the natural default. A remote invocation can spend meaningful time crossing a network, waiting in another process, doing work, and crossing the network again. Distribution changed the cost of waiting. So even before anything fails, the network has already leaked into the programming model. Then something fails. The Timeout Is Where the Illusion Gets Cursed The client sends a request. Nothing comes back. So it waits. Eventually it times out. What happened? That sounds like a simple question. It is not. Perhaps the request packet was lost before reaching the server. Perhaps the request reached the server, but the acknowledgement was lost. Perhaps the server executed the procedure and the reply was lost. Perhaps the network became unavailable. Perhaps the server process failed. Perhaps the entire server machine failed. From the client’s point of view, several very different realities can collapse into exactly the same observation: No reply arrived. This does not happen in the same way with an ordinary local procedure call. The network has introduced ambiguity. And ambiguity is where “remote is just like local” finally stops being an innocent simplification. Fine. Retransmit It. For ordinary communication failures, there are reasonable techniques. Give requests unique identifiers or sequence numbers. Use acknowledgements. Detect duplicates. If a message appears to have been lost, retransmit it. This works beautifully for a lost packet. Client sends request. No acknowledgement. Timeout. Client sends request again. Server receives it. Reply arrives. _Problem solved. _Except there is another possible history. Client sends request. Server receives it. Server performs the operation. The acknowledgement or reply disappears. Client times out. Client sends the request again. Now the server has seen the same operation twice. For something harmless, perhaps that does not matter. For an operation with side effects, it matters quite a lot. Imagine the remote procedure is: TransferMoney(100) The client did not ask: MaybeTransferMoneySomeNumberOfTimes(100) Retries make the transport more reliable. They can simultaneously make the operation less obviously correct. This is why duplicate detection exists. A server can recognize an identifier it has already processed and avoid executing the same logical request again. ONC RPC similarly uses transaction IDs to match requests and replies and describes retaining those IDs as a way of obtaining a degree of execute-at-most-once behaviour. (RFC Editor) The retry mechanism is not merely recovering lost packets. It is trying to recover the meaning of the original invocation. Did the Server Run It? Now we reach the unpleasant question. The client sends a request. The server executes it. Before the client receives the reply, communication disappears. What should the client conclude? The tempting answer is: The call failed. But what exactly does failed mean? It certainly means the client did not receive a successful result. It does not necessarily mean the server did not perform the operation. That is the hidden difference. The client knows what it observed. It does not automatically know what happened on the other machine. Even with a reliable transport such as TCP, receiving a reply allows the caller to know the remote operation executed, but receiving no reply does not prove that it