Key Question

How does Redis Cluster distribute data across nodes, and why did Redis choose 16384 fixed slots over consistent hashing?

Deep Dive

Redis Cluster is not a single system — it is a distributed sharding layer that sits on top of Redis’s in-memory single-node capabilities. Unlike Cassandra or Riak (which were distributed from day one), Redis started as a single-machine cache and had distribution added later. This history is visible in every design decision.

The 16384 Hash Slots

Instead of consistent hashing (Dynamo, Cassandra, Riak) or a flat key range (Bigtable), Redis divides the key space into 16384 fixed slots. A key’s slot is determined by CRC16(key) mod 16384. Each node in the cluster claims ownership of a contiguous range of slots.

function hashSlot(key: string): number {
  const tagMatch = key.match(/\{(.+?)\}/)
  const effective = tagMatch ? tagMatch[1] : key
  return crc16(Buffer.from(effective)) % 16384
}

Why 16384? The Redis author (antirez) explained: 16384 slots can be represented efficiently in gossip messages (2 KB bitmap), which keeps cluster bus messages small. Consistent hashing would require more complex reconciliation.

Hash Tags

Redis Cluster supports hash tags: if a key contains {...}, only the content inside the braces is hashed. This is a deliberate escape hatch — it lets developers force related keys (e.g., user:{42}:profile, user:{42}:orders) onto the same node so they can be accessed in a single atomic operation. Redis trades pure distribution for practical multi-key operations.

Asynchronous Replication

Each master has 0 to N replicas. Replication is asynchronous by design — the master does not wait for a replica to acknowledge before responding to the client. This means:

• Writes are fast (single in-memory operation + send to replica in background)
• Failover can lose data (the last writes acknowledged to the client may not have reached the replica)

This is a deliberate trade-off. Redis prioritizes latency and availability over durability and consistency. A Redis Cluster that loses a few recent writes during failover is operating as designed.

Key Takeaways

• Redis uses 16384 fixed hash slots (not consistent hashing) for simplicity and compact gossip messages.
• Hash tags allow grouping keys for atomic multi-key operations.
• Asynchronous replication makes writes fast but means failover can lose data.
• Redis Cluster sacrifices consistency for performance — it is a PA/EL system in PACELC terms.

Full Source

View or download the complete implementation: redis-cluster.ts

Exercises

1. Why did Redis choose 16384 slots instead of 65536 or 4096?
2. What problem do hash tags solve? Give an example where they are necessary.
3. During a master failure, a replica is promoted. A client writes a key that was routed to the old master just before the failure. What happens to that write?
4. Compare Redis Cluster’s slot-based routing with Cassandra’s consistent hashing. What are the operational differences?

👁️ View Solutions

1. 16384 is the sweet spot: 4096 would cause too much data movement on topology changes; 65536 would make gossip messages too large (8 KB bitmap per node). With 16384, a slot bitmap fits in 2 KB, keeping cluster bus messages lean.
2. Hash tags force keys with the same tag onto the same node, enabling MGET/MSET/multi-key transactions. Example: {shard:1}:user and {shard:1}:posts are always on the same node. Without hash tags, a MGET on related keys would scatter across nodes and fail.
3. The write is lost. Since replication is asynchronous, the replica never received the write. After promotion, the new master serves stale data. This is Redis Cluster’s fundamental consistency trade-off.
4. Redis uses fixed-size slot ranges (contiguous blocks), making slot migration predictable and manual (via CLUSTER SETSLOT). Cassandra uses hundreds of virtual nodes per machine, making rebalancing automatic but less predictable. Redis requires manual resharding; Cassandra handles it transparently.

✏️ Exercises

Redis Cluster — Exercises

Exercise 1

You have a 6-node Redis Cluster (3 masters, 3 replicas). A master node handling slots 5461–10922 crashes. Its replica is promoted. What keys are lost?

Exercise 2

Given the key set: user:100, {shard:a}:profile, order:{2024}:001, session:abc, compute the hash slot for each using CRC16 mod 16384. Which keys would land on the same node if you use hash tags?

Exercise 3

Redis Cluster’s gossip protocol requires each node to maintain its own epoch (configuration version). When a new node joins, its epoch is 0. The cluster leader assigns it a higher epoch. Why is this epoch necessary? (Hint: think about stale routing tables.)

Exercise 4

Compare the slot bitmap approach (Redis) with the token ring approach (Cassandra). Which requires more metadata to route a request? Which handles node addition more smoothly?

👁️ View Solutions

1. Only the keys that were written to the crashed master but not yet replicated to the replica. Since replication is asynchronous, the window is typically 1–100 milliseconds of writes. The keys on the surviving masters are unaffected.

2. Hash slot computation:

   • CRC16("user:100") mod 16384 = slot X
   • CRC16("shard:a") mod 16384 = slot Y (only “shard:a” due to {})
   • CRC16("2024") mod 16384 = slot Z (only “2024” due to {})
   • CRC16("session:abc") mod 16384 = slot W
   • {shard:a}:profile and {shard:a}:name would share the same slot. order:{2024}:001 and order:{2024}:002 would share a different slot.

3. The epoch prevents stale routing information from overwriting fresh data. If a partitioned node wakes up with an old epoch and broadcasts its slot map, nodes with a higher epoch know to ignore it. This is functionally identical to Raft’s term number.

4. Redis uses a fixed-size 16384-entry array — O(1) lookup, 2 KB per node. Cassandra’s token ring uses a sorted list of tokens with binary search — O(log N) lookup. Redis is simpler at the cost of manual rebalancing; Cassandra is more complex but handles additions and removals automatically.