Distributed & Decentralized Systems Curriculum
Consistency Trade offs · CRDTs

Key Question

How do we design data structures that can be replicated across multiple nodes, updated concurrently, and still merge into a single consistent state?

Deep Dive

The crdt.ts file implements four CRDT variants in TypeScript. Each demonstrates a different strategy for achieving eventual consistency without conflicts. There is no “last writer wins” or “manual merge” — the math guarantees convergence.

1. G-Counter: The Grow-Only Counter

A G-Counter only supports increment. Each node tracks its own contributions in a map: { nodeId: count }.

class GCounter {
  private counts: Map<number, number> = new Map()

  constructor(private nodeId: number) {}

  increment(by: number = 1) {
    this.counts.set(this.nodeId,
      (this.counts.get(this.nodeId) ?? 0) + by)
  }

  value(): number {
    let total = 0
    for (const v of this.counts.values()) total += v
    return total
  }

  merge(other: GCounter): void {
    // Take the max value for each node — this is the
    // join semilattice operation (commutative, associative, idempotent)
    for (const [key, value] of other.counts) {
      this.counts.set(key,
        Math.max(this.counts.get(key) ?? 0, value))
    }
  }
}

The value is the sum of all entries. The merge operation takes the maximum value for each key. Since counters only grow, taking the max is safe. Mathematically, this is a join semilattice: commutative, associative, and idempotent.

2. PN-Counter: Going Negative

A G-Counter can only increase. A PN-Counter solves this by using two G-Counters: one for increments (P) and one for decrements (N). value = sum(P) - sum(N):

class PNCounter {
  private p: GCounter  // increments
  private n: GCounter  // decrements

  constructor(nodeId: number) {
    this.p = new GCounter(nodeId)
    this.n = new GCounter(nodeId)
  }

  increment(by: number = 1) { this.p.increment(by) }
  decrement(by: number = 1) { this.n.increment(by) }

  value(): number { return this.p.value() - this.n.value() }

  merge(other: PNCounter): void {
    this.p.merge(other.p)  // Both G-Counters converge independently
    this.n.merge(other.n)  // so their difference also converges
  }
}

When you increment, you increment P. When you decrement, you increment N. Since both P and N are G-Counters, they converge independently, and therefore the difference also converges.

3. LWW-Register: Time as a Tiebreaker

An LWW-Register stores a value plus a timestamp. On merge, the value with the highest timestamp wins:

class LWWRegister<T> {
  private value: T | null = null
  private timestamp: number = 0
  private nodeId: number

  constructor(nodeId: number) { this.nodeId = nodeId }

  assign(newValue: T, clock: number) {
    this.value = newValue
    this.timestamp = clock
  }

  merge(other: LWWRegister<T>): void {
    // Compare (timestamp, nodeId) lexicographically.
    // Node ID breaks ties deterministically.
    if (other.timestamp > this.timestamp ||
       (other.timestamp === this.timestamp &&
        other.nodeId > this.nodeId)) {
      this.value = other.value
      this.timestamp = other.timestamp
    }
  }
}

If timestamps tie, we use the node ID as a deterministic tiebreaker. This ensures convergence even with concurrent writes. The weakness: data loss. If two nodes write concurrently, one value is silently discarded.

4. OR-Set: The Observed-Remove Set

The OR-Set solves the “add then remove” problem that plagues naive replicated sets. Every “add” creates a unique tag. A “remove” only removes the tags the remover has observed:

class ORSet<T> {
  // Map from element → Set of unique tags
  private addSet: Map<string, Set<string>> = new Map()
  private removeSet: Map<string, Set<string>> = new Map()

  add(element: T) {
    const key = JSON.stringify(element)
    if (!this.addSet.has(key)) this.addSet.set(key, new Set())
    this.addSet.get(key)!.add(this.makeTag()) // unique tag per add
  }

  remove(element: T) {
    const key = JSON.stringify(element)
    const tags = this.addSet.get(key)
    if (tags) {
      if (!this.removeSet.has(key)) this.removeSet.set(key, new Set())
      // Only remove the tags we have OBSERVED
      for (const tag of tags) this.removeSet.get(key)!.add(tag)
    }
  }

  has(element: T): boolean {
    const key = JSON.stringify(element)
    const addTags = this.addSet.get(key)
    const removeTags = this.removeSet.get(key)
    if (!addTags) return false
    if (!removeTags) return true
    // Visible if ANY add tag is NOT in the tombstone
    for (const tag of addTags)
      if (!removeTags.has(tag)) return true
    return false
  }

  merge(other: ORSet<T>): void {
    // Union of both add sets and remove sets
    for (const [key, tags] of other.addSet) { /* union add */ }
    for (const [key, tags] of other.removeSet) { /* union remove */ }
  }
}

The OR-Set guarantee: If Node 1 adds “banana” (tag t1) and Node 2 concurrently removes “banana” (only removing tag t2 which it observed), then tag t1 survives the merge. You can only remove what you have observed.

Key Takeaways

  • G-Counter/PN-Counter are “state-based” CRDTs: the entire state is exchanged and merged.
  • LWW-Register uses timestamps for conflict resolution but loses concurrent writes silently.
  • OR-Set preserves causal history: an element added independently survives concurrent removes at other nodes.
  • All four types are commutative: the order of merges doesn’t matter, guaranteeing convergence.

Running the Code

cd lessons/02-Consistency-Trade-offs/05-CRDTs/
npx ts-node crdt.ts

Full Source

View or download the complete implementation: crdt.ts

Exercises

  1. Why can’t a G-Counter support decrement? What would break?
  2. In an LWW-Register, why do we compare (timestamp, nodeId) instead of just timestamp?
  3. Why does “banana” survive in the OR-Set simulation even though Node 2 removed it?

👁️ View Solutions

  1. A G-Counter merge takes the max of per-node counts. Decrement violates the grow-only property — a decrement would be lost if the other node had a higher value, causing the counter to “undecrement” after a merge.
  2. If two writes happen at the exact same timestamp (e.g., same logical clock value on different nodes), the system needs a tiebreaker. Appending the node ID creates a total order out of the partial order.
  3. In the simulation, Node 1 added “banana” (tag t1) and Node 2 added “banana” (tag t2) independently. Node 2 removed “banana” but only removed tag t2 (its own tag). Tag t1 (from Node 1) was never observed by Node 2, so it survives the merge.

✏️ Exercises

Exercises: CRDTs

Exercise 1: G-Counter Merge

A G-Counter has 3 replicas. Replica A’s state is [5, 0, 0]. Replica B’s state is [0, 3, 2].

  1. What is the state of each replica after they merge (A with B, B with A)?
  2. What is the final counter value?
  3. Now replica C (state [1, 1, 1]) joins the merge. What is the final result?

Exercise 2: G-Counter Limitations

  1. Explain in one sentence why a G-Counter cannot represent decrements.
  2. Could you build a G-Counter where decrement is allowed by subtracting from the local slot? What goes wrong?
  3. What mathematical property of the merge function would break if you allowed arbitrary subtraction?

Exercise 3: OR-Set Concurrent Operations

Three replicas (A, B, C) start with an empty OR-Set.

  • Replica A: adds “apple” (tag = a1), adds “banana” (tag = a2)
  • Replica B: adds “apple” (tag = b1), removes “banana” (removes a2 — B observed A’s add)
  • Replica C: removes “apple” (removes a1, b1 — C observed both)

After all replicas merge:

  1. Is “apple” in the set? Why or why not?
  2. Is “banana” in the set? Why or why not?
  3. How many unique tags are in the final set?

Exercise 4: CRDT vs Consensus

An e-commerce platform uses an inventory CRDT (PN-Counter) to track stock across regions. The current count for item X is 5.

  1. Two regions both sell one unit simultaneously (decrement by 1 each). What does the counter show after merge?
  2. The actual physical inventory is 5 units. After the decrements above, you sell 3 more units in region A and 2 more in region B. A customer checks the count and sees 3. They buy 3 — but the warehouse only has 2 left. How could this over-selling happen?
  3. Would Raft prevent this problem? What trade-off does it introduce?
👁️ View Solutions

Solutions: CRDTs

Exercise 1: G-Counter Merge

  1. After A merges with B:

    • A receives [0, 3, 2] from B → elementwise max of [5, 0, 0] and [0, 3, 2] = [5, 3, 2]
    • B receives [5, 0, 0] from A → elementwise max of [0, 3, 2] and [5, 0, 0] = [5, 3, 2]
    • Both converge to [5, 3, 2]
  2. Final counter value: 5 + 3 + 2 = 10

  3. C joins: max of [5, 3, 2] and [1, 1, 1]:

    • max(5, 1) = 5
    • max(3, 1) = 3
    • max(2, 1) = 2
    • Result: [5, 3, 2]C’s values were all lower, so they don’t change the outcome.

Exercise 2: G-Counter Limitations

  1. A G-Counter’s merge function is elementwise max — taking the maximum of two values can never produce a smaller number, so you can never reduce any slot.

  2. If you subtract from your local slot, the elementwise max during merge would take the larger previous value from another replica. Your decrement would be silently lost. For example: A has [5, 0, 0], A decrements to [4, 0, 0], then merges with B’s [5, 0, 0] (which hadn’t seen the decrement). Result: [5, 0, 0] — the decrement disappeared.

  3. Elementwise max is not invertible. If you allow subtraction, the merge function would need to decrease a value, which max cannot do. The lattice property (join = least upper bound) only works with monotonic increases.


Exercise 3: OR-Set Concurrent Operations

  1. Is “apple” in the set? Yes. C removed tags a1 and b1. But A had already added “apple” with a1, and B had added with b1. Both are removed by C’s operation. However… wait, let’s trace carefully:

    • A adds “apple” with tag a1. A adds “banana” with tag a2.
    • B adds “apple” with tag b1. B removes “banana” — removes tag a2.
    • C removes “apple” — removes tags a1 and b1 (that C observed).

    After merge: A has {(“apple”, a1), (“banana”, a2)} ∪ {(“apple”, b1)} — but a1 and b1 were removed by C. However, the merge is a union of internal sets. C’s state has { } (it removed everything it knew about). A has {(“apple”, a1), (“banana”, a2), (“apple”, b1)} only if B’s adds propagated. But B also sent its adds.

    Actually, OR-Set merge is set union of (element, tag) pairs. C’s remove of tags a1 and b1 means those tags are excluded from C’s internal set. But when A merges with C, A still has a1 and b1 (A received b1 from B). So the union brings them back. Wait — that breaks the semantics.

    Correction: In an OR-Set, when you remove, you add the removed tags to a “tombstone” set (or equivalently, the internal set is {(elem, tag)} and remove means “remove all observed (elem, tag) pairs”). But the merge is union — if another replica still has those tags, they come back.

    This is why real OR-Sets use “observed-remove” with version vectors or tombstones. The answer depends on implementation. For the standard OR-Set:

    After all merges (union of all (e,t) pairs minus removed pairs):

    • Banana’s a2 tag is removed by B, and only A had it. Gone.
    • Apple’s a1 is removed by C, but B has b1 (different tag) — wait, a1 is explicitly removed, but a1 was from A. B had b1, not a1.

    Let me retrace properly: after full sync:

    • All tags: a1 (“apple”), a2 (“banana”), b1 (“apple”)
    • Removed tags: a2 (by B), a1 and b1 (by C)
    • Surviving tags: none — all three tags are in the remove set.

    Final set: empty {}.

    This is actually a problem with basic OR-Sets — concurrent adds and removes can delete everything. This is why production OR-Sets often use version vectors or “add wins” semantics more carefully.

  2. Is “banana” in the set? No — tag a2 was removed by B, and no other replica added “banana”. a2 is in the tombstone set.

  3. Unique tags surviving: 0 (everything was removed).

    Key insight: This exercise shows a weakness of basic OR-Sets — if a remove operation observes all tags and removes them, the element disappears. Production OR-Sets add version vectors to track causality so that concurrent adds survive.


Exercise 4: CRDT vs Consensus

  1. After both regions decrement: Both decrements are recorded. Region A’s N-counter has [1, 0], Region B’s has [0, 1]. After merge: P = [0, 0], N = [1, 1]. Value = 0 - (1 + 1) = -2. Wait — the initial value was 5, so P should track the initial value. Let me reconsider.

    Initial state (both regions): P = [5, 5], N = [0, 0] (each region’s P-counter has “5” for the initial stock). If each sells 1: region A decrements → N_A = [1, 0], region B → N_B = [0, 1].

    After merge: P = [5, 5], N = [1, 1]. Value = (5+5) - (1+1) = 10 - 2 = 8. That’s wrong — we only had 5 units initially. This reveals a problem: each region independently initialized P to 5. The initialization is not coordinated.

    The real answer: A PN-Counter for inventory must be initialized on one replica and propagated. If both regions independently set P to 5, the merge results in P = max(5,5) = 5 per slot, sum(P) = 10 — double counting. This is why initial state needs coordination.

    But ignoring the initialization issue: after both decrements, the counter shows 3 (5 - 1 - 1 = 3).

  2. Over-selling: The counter shows 3 (a stale or concurrent read), but in reality, by the time that customer checks, 5 more units have been sold across both regions (3 + 2). The counter eventually converges to show -2, but during the sync delay, the customer saw 3 and bought 3. The system sold 10 units of a 5-unit inventory.

  3. Would Raft prevent this? Yes — Raft provides strong consistency. All writes go through a leader, so inventory is always accurate. Trade-off: Writes are slower (require majority ack), and if the leader is in a different region, every write incurs cross-region latency. During a network partition, the minority partition cannot sell anything. CRDTs sacrifice accuracy (stale reads) for availability (local writes always succeed).