Key Question
How does a Remote Procedure Call make a function on another machine look like a local call?
Deep Dive
A Remote Procedure Call (RPC) is the distributed systems equivalent of a function call. You call getUser(id) and it feels local β but behind the scenes, the call crosses the network, executes on another machine, and returns the result. The entire mechanism is designed to hide the network.
The Call Chain
Here is what happens when a client calls add(3, 4) on a remote server:
Client Machine Server Machine
βββββββββββββββββββ ββββββββββββββββββββ
β Client Code β β Server Code β
β add(3, 4) β β int add(a,b) { β
β β β β return a+b; β
β βΌ β β } β
β Client Stub β β β² β
β marshals args β β β β
β βββΊ [3, 4] β β Server Stub β
β β β β unmarshals args β
β βΌ β β calls add(3,4) β
β Transport β β β β
β (TCP/UDP) β β Transport β
β β β β β β
βββββββββΌβββββββββββ βββββββββΌβββββββββββ
β β
ββββββββ NETWORK βββββββββββββ
Request: [3, 4] ββββββββββββββΊ
ββββββββββββββ Response: 7
Step by Step
-
Client calls local stub β The client invokes
add(3, 4)on a local function. It doesnβt know the implementation is remote. -
Client stub marshals arguments β The stub serializes the arguments
3and4into a flat byte buffer (marshaling). It also packs a call ID so the server knows which function to invoke. -
Transport layer sends β The marshaled bytes are sent over the network (usually TCP, sometimes UDP for speed).
-
Server stub receives β The serverβs transport layer hands bytes to the server stub.
-
Server stub unmarshals and calls β The stub deserializes
3and4, looks up the function by call ID, and invokes the actualadd(3, 4)on the server. -
Result travels back β The return value
7is marshaled and sent back to the client. The client stub unmarshals it and returns to the caller.
The RPC Functional Model
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
β Caller ββββΊβ Client ββββΊβ Server ββββΊβ Callee β
β (client) β β Stub β β Stub β β (server) β
ββββββββββββ ββββββββββββ ββββββββββββ ββββββββββββ
The client stub (proxy) and server stub (skeleton) are auto-generated from an interface definition. The developer writes code against the stub as if it were local.
RPC vs REST
| Aspect | RPC | REST |
|---|---|---|
| Orientation | Action | Resource |
| Call style | createUser(name) | POST /users |
| HTTP verbs | Usually POST | GET, POST, PUT, DELETE |
| Caching | Harder | Native (HTTP cache) |
| Coupling | Tight (client knows procedure names) | Loose (client knows resource paths) |
RPC is action-oriented: chargeCard(), sendEmail(), computeScore(). REST is resource-oriented: POST /payments, POST /emails, GET /scores/42. Choose RPC when you have distinct actions (not CRUD). Choose REST when youβre primarily creating, reading, updating, and deleting resources.
Check Your Understanding
-
What would break if the client stub and server stub were generated from different interface definitions?
-
Why is marshaling different from just writing raw bytes to a socket?
-
REST APIs use standard HTTP methods. Does this give them any practical advantage over RPC?
The βSo What?β
RPC is the backbone of most distributed systems β gRPC, Java RMI, Thrift, CORBA, JSON-RPC, XML-RPC. Understanding the call chain helps you debug performance issues (where is time spent? marshaling? network? server execution?), reason about failure modes (what happens if the server crashes after step 5 but before step 6?), and choose the right communication pattern for your system.
βοΈ Exercises
Exercises: Architectural Models & RPC
Exercise 1: Architecture Choice
A team is building a file-sharing application for 100,000 users. Files are read-heavy (mostly downloads), users are geographically distributed, and there is no budget for centralized infrastructure.
- Which architectural model (client-server, P2P, multi-tier, hybrid) would you recommend?
- What specific design decisions would you make to handle the read-heavy workload?
- What are the top three problems youβd need to solve?
Exercise 2: RPC Call Failure Analysis
A client calls deductBalance(userID: "u42", amount: 50.00) via RPC. The client stub sends the request to the server. For each scenario below, state what happens and whether the clientβs balance is correct:
- The client stub marshals the request and sends it. The server receives it, unmarshals, calls the function (which deducts $50 from the DB), but the server crashes before sending the response.
- The server receives the request, processes it, and sends the response. The response is lost in the network. The client times out and throws an exception.
- The client sends the request. The server processes it and sends the response. The client receives the response. Everything works β but the network duplicates the request and the server processes it twice.
Exercise 3: Protobuf Field Evolution
You have this protobuf schema deployed in production:
message Order {
string order_id = 1;
string user_id = 2;
float total = 3;
}
You want to add a string coupon_code = 4 field. Some old server instances still running donβt know about field 4.
- Will old servers crash when they receive a message with
coupon_codeset? - A client built from the old schema processes a message that has
coupon_code. What does the client see? - What happens if you later delete field 3 (
total) and reuse its number for a newint64 total_cents = 3?
ποΈ View Solutions
Solutions: Architectural Models & RPC
Exercise 1: Architecture Choice
-
Recommended model: Hybrid with P2P as the primary model β users share files directly with each other β plus a small set of super-peers (or a lightweight tracker) for discovery.
- Pure P2P (BitTorrent-style) handles read-heavy workloads naturally: popular files are replicated across many peers, distributing the download load.
- A tracker (a lightweight centralized component) solves the discovery problem β where to find each file.
- A small number of βseedβ servers can ensure unpopular files remain available (no single point of failure because seeds are optional).
-
Design decisions:
- Chunk files into pieces so peers can download different chunks from different peers in parallel.
- Use content-addressed storage (hash of the file content as the identifier) to verify integrity.
- Implement a tit-for-tat incentive mechanism (you share, you get faster downloads).
-
Top three problems:
- Discovery: How do peers find each other and learn which files are available? (Tracker nodes + DHT)
- Churn: Peers join and leave constantly. How do you maintain availability of rare files? (Replication factor, redundancy)
- Trust: How do you prevent peers from serving corrupted data? (Content hashing, cryptographic verification)
Exercise 2: RPC Call Failure Analysis
-
Correct? β No. The server deducted $50 but crashed before the client got confirmation. The client assumes the call failed and may retry. You now have a duplicate deduction (or at least a $50 mismatch unless the operation is idempotent). This is the classic βat-most-once vs at-least-onceβ dilemma. Solution: make
deductBalanceidempotent (using a request ID or idempotency key). -
Correct? β The server did deduct $50. The client got an exception and doesnβt know whether the deduction happened. This is the exactly-once is impossible problem in distributed systems. The client must check the balance or retry with idempotency.
-
Correct? β No. The balance is deducted twice β $100 total instead of $50. The server processed the request twice because the transport layer delivered a duplicate. Solution: deduplication at the server (track recently seen request IDs).
Key insight: In all three cases, the client cannot trivially know the correct balance without additional mechanisms (idempotency keys, at-least-once delivery with dedup, transactional outboxes).
Exercise 3: Protobuf Field Evolution
-
Will old servers crash? No. Protobuf is designed for forward compatibility. Old servers that donβt know about field 4 will simply ignore the unknown bytes. The message is self-describing enough that unknown fields are skipped during deserialization.
-
What does the old client see? The
coupon_codefield will be absent (default empty string). The client seesorder.coupon_code == "". The data is not lost β if the client re-serializes the message, the unknown bytes for field 4 are preserved and passed through. -
What happens if you delete field 3 and reuse its number? Disaster. Old servers still running will interpret the new
total_centsfield as the oldtotalfield and read garbage. Never reuse a field number. Instead, mark the field asreserved 3;in the new schema β this prevents accidental reuse.