How distributed locks actually work in production

We had a payment processing bug that cost us about $4,000 in duplicate charges before we caught it. The root cause? A distributed lock implementation that worked perfectly in testing and failed subtly in production.

Let me save you the 3am phone call.

The naive Redis lock

Most teams implement this the same way: SET key value NX EX 30. Set a key if it doesn't exist, expire it in 30 seconds. Release it by deleting the key. Simple, right?

Wrong. The problem is the release step. If you just DEL key, you might delete a lock that's not yours. Here's the scenario: your lock expires (because your process got slow), another process acquires it, and then your original process wakes up and deletes it. Now nobody has the lock, but both processes think they own the critical section.

The right way: check-then-delete atomically

You need a Lua script to make the release atomic:

if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end

The value you set is a UUID unique to your lock acquisition. When you release, you verify it's still your UUID before deleting. Since Lua scripts run atomically in Redis, you can't have a race between the check and the delete.

Clock drift is still your enemy

Even with the Lua script, you're not done. Redis TTL-based locks assume your clock and Redis's clock agree. In reality, clocks drift. VMs get paused. GC stops the world for 500ms. Your 30-second lock might effectively be a 5-second lock in a degraded environment.

The real answer for high-stakes locking is Redlock across multiple independent Redis nodes, or better yet, using something like etcd or ZooKeeper that's built for distributed coordination. Redis is a cache. It's great at a lot of things. Distributed consensus isn't really one of them.

If money moves on it, don't use a single Redis instance for locking.

We switched to idempotency keys at the database level for payments. The lock is now just an optimization to reduce duplicate attempts, not a correctness guarantee. Sleep better at night.