Distributed & Decentralized Systems Curriculum
Time Causality Β· Architectural Models RPC

Key Question

How does a Remote Procedure Call make a function on another machine look like a local call?

Deep Dive

A Remote Procedure Call (RPC) is the distributed systems equivalent of a function call. You call getUser(id) and it feels local β€” but behind the scenes, the call crosses the network, executes on another machine, and returns the result. The entire mechanism is designed to hide the network.

The Call Chain

Here is what happens when a client calls add(3, 4) on a remote server:

  Client Machine                 Server Machine
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚   Client Code    β”‚           β”‚   Server Code     β”‚
  β”‚   add(3, 4)      β”‚           β”‚   int add(a,b) {  β”‚
  β”‚       β”‚          β”‚           β”‚     return a+b;   β”‚
  β”‚       β–Ό          β”‚           β”‚   }               β”‚
  β”‚   Client Stub    β”‚           β”‚       β–²           β”‚
  β”‚   marshals args  β”‚           β”‚       β”‚           β”‚
  β”‚   ──► [3, 4]     β”‚           β”‚   Server Stub     β”‚
  β”‚       β”‚          β”‚           β”‚   unmarshals args β”‚
  β”‚       β–Ό          β”‚           β”‚   calls add(3,4)  β”‚
  β”‚   Transport      β”‚           β”‚       β”‚           β”‚
  β”‚   (TCP/UDP)      β”‚           β”‚   Transport       β”‚
  β”‚       β”‚          β”‚           β”‚       β”‚           β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜           β””β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
          β”‚                             β”‚
          └───────  NETWORK β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
          Request: [3, 4]  ─────────────►
          ◄─────────────  Response: 7

Step by Step

  1. Client calls local stub β€” The client invokes add(3, 4) on a local function. It doesn’t know the implementation is remote.

  2. Client stub marshals arguments β€” The stub serializes the arguments 3 and 4 into a flat byte buffer (marshaling). It also packs a call ID so the server knows which function to invoke.

  3. Transport layer sends β€” The marshaled bytes are sent over the network (usually TCP, sometimes UDP for speed).

  4. Server stub receives β€” The server’s transport layer hands bytes to the server stub.

  5. Server stub unmarshals and calls β€” The stub deserializes 3 and 4, looks up the function by call ID, and invokes the actual add(3, 4) on the server.

  6. Result travels back β€” The return value 7 is marshaled and sent back to the client. The client stub unmarshals it and returns to the caller.

The RPC Functional Model

  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚  Caller  │──►│  Client  │──►│  Server  │──►│  Callee  β”‚
  β”‚ (client) β”‚   β”‚  Stub    β”‚   β”‚  Stub    β”‚   β”‚ (server) β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The client stub (proxy) and server stub (skeleton) are auto-generated from an interface definition. The developer writes code against the stub as if it were local.

RPC vs REST

AspectRPCREST
OrientationActionResource
Call stylecreateUser(name)POST /users
HTTP verbsUsually POSTGET, POST, PUT, DELETE
CachingHarderNative (HTTP cache)
CouplingTight (client knows procedure names)Loose (client knows resource paths)

RPC is action-oriented: chargeCard(), sendEmail(), computeScore(). REST is resource-oriented: POST /payments, POST /emails, GET /scores/42. Choose RPC when you have distinct actions (not CRUD). Choose REST when you’re primarily creating, reading, updating, and deleting resources.

Check Your Understanding

  1. What would break if the client stub and server stub were generated from different interface definitions?

  2. Why is marshaling different from just writing raw bytes to a socket?

  3. REST APIs use standard HTTP methods. Does this give them any practical advantage over RPC?

The β€œSo What?”

RPC is the backbone of most distributed systems β€” gRPC, Java RMI, Thrift, CORBA, JSON-RPC, XML-RPC. Understanding the call chain helps you debug performance issues (where is time spent? marshaling? network? server execution?), reason about failure modes (what happens if the server crashes after step 5 but before step 6?), and choose the right communication pattern for your system.


✏️ Exercises

Exercises: Architectural Models & RPC

Exercise 1: Architecture Choice

A team is building a file-sharing application for 100,000 users. Files are read-heavy (mostly downloads), users are geographically distributed, and there is no budget for centralized infrastructure.

  1. Which architectural model (client-server, P2P, multi-tier, hybrid) would you recommend?
  2. What specific design decisions would you make to handle the read-heavy workload?
  3. What are the top three problems you’d need to solve?

Exercise 2: RPC Call Failure Analysis

A client calls deductBalance(userID: "u42", amount: 50.00) via RPC. The client stub sends the request to the server. For each scenario below, state what happens and whether the client’s balance is correct:

  1. The client stub marshals the request and sends it. The server receives it, unmarshals, calls the function (which deducts $50 from the DB), but the server crashes before sending the response.
  2. The server receives the request, processes it, and sends the response. The response is lost in the network. The client times out and throws an exception.
  3. The client sends the request. The server processes it and sends the response. The client receives the response. Everything works β€” but the network duplicates the request and the server processes it twice.

Exercise 3: Protobuf Field Evolution

You have this protobuf schema deployed in production:

message Order {
  string order_id = 1;
  string user_id = 2;
  float total = 3;
}

You want to add a string coupon_code = 4 field. Some old server instances still running don’t know about field 4.

  1. Will old servers crash when they receive a message with coupon_code set?
  2. A client built from the old schema processes a message that has coupon_code. What does the client see?
  3. What happens if you later delete field 3 (total) and reuse its number for a new int64 total_cents = 3?
πŸ‘οΈ View Solutions

Solutions: Architectural Models & RPC

Exercise 1: Architecture Choice

  1. Recommended model: Hybrid with P2P as the primary model β€” users share files directly with each other β€” plus a small set of super-peers (or a lightweight tracker) for discovery.

    • Pure P2P (BitTorrent-style) handles read-heavy workloads naturally: popular files are replicated across many peers, distributing the download load.
    • A tracker (a lightweight centralized component) solves the discovery problem β€” where to find each file.
    • A small number of β€œseed” servers can ensure unpopular files remain available (no single point of failure because seeds are optional).
  2. Design decisions:

    • Chunk files into pieces so peers can download different chunks from different peers in parallel.
    • Use content-addressed storage (hash of the file content as the identifier) to verify integrity.
    • Implement a tit-for-tat incentive mechanism (you share, you get faster downloads).
  3. Top three problems:

    • Discovery: How do peers find each other and learn which files are available? (Tracker nodes + DHT)
    • Churn: Peers join and leave constantly. How do you maintain availability of rare files? (Replication factor, redundancy)
    • Trust: How do you prevent peers from serving corrupted data? (Content hashing, cryptographic verification)

Exercise 2: RPC Call Failure Analysis

  1. Correct? β€” No. The server deducted $50 but crashed before the client got confirmation. The client assumes the call failed and may retry. You now have a duplicate deduction (or at least a $50 mismatch unless the operation is idempotent). This is the classic β€œat-most-once vs at-least-once” dilemma. Solution: make deductBalance idempotent (using a request ID or idempotency key).

  2. Correct? β€” The server did deduct $50. The client got an exception and doesn’t know whether the deduction happened. This is the exactly-once is impossible problem in distributed systems. The client must check the balance or retry with idempotency.

  3. Correct? β€” No. The balance is deducted twice β€” $100 total instead of $50. The server processed the request twice because the transport layer delivered a duplicate. Solution: deduplication at the server (track recently seen request IDs).

Key insight: In all three cases, the client cannot trivially know the correct balance without additional mechanisms (idempotency keys, at-least-once delivery with dedup, transactional outboxes).


Exercise 3: Protobuf Field Evolution

  1. Will old servers crash? No. Protobuf is designed for forward compatibility. Old servers that don’t know about field 4 will simply ignore the unknown bytes. The message is self-describing enough that unknown fields are skipped during deserialization.

  2. What does the old client see? The coupon_code field will be absent (default empty string). The client sees order.coupon_code == "". The data is not lost β€” if the client re-serializes the message, the unknown bytes for field 4 are preserved and passed through.

  3. What happens if you delete field 3 and reuse its number? Disaster. Old servers still running will interpret the new total_cents field as the old total field and read garbage. Never reuse a field number. Instead, mark the field as reserved 3; in the new schema β€” this prevents accidental reuse.