Concurrency: From Race Conditions to Distributed Locks

Suppose we have an ecommerce app with an endpoint to buy a product:

POST /buy-product

There is only one item left, and two users try to buy it at almost the same time.

Both requests check the stock, both find that the product is available, and both create an order. Now we have confirmed two orders for one item.

The code might work perfectly when we test one request at a time. The problem appears when requests overlap and access the same data.

In this article, we’ll understand:

• processes, threads, concurrency, and parallelism
• race conditions and critical sections
• mutexes, deadlocks, and semaphores
• atomic operations and compare-and-swap
• thread pools, event loops, and queues
• database locking
• distributed locks and fencing tokens
• idempotency

Processes and threads

A process is a running instance of a program. It generally has its own memory space. Two separate backend processes don’t automatically share their variables.

A thread is a unit of execution inside a process. Threads in the same process share its memory, although each thread also has its own execution stack.

For example, two threads might both access the same in-memory inventory object. That makes sharing data convenient, but we need to coordinate how they use it.

Concurrency vs parallelism

Concurrency means multiple tasks are in progress during overlapping periods. Parallelism means multiple tasks are actually executing at the same instant.

On a single CPU core, the system can switch between tasks:

Task A -> Task B -> Task A -> Task B

On multiple cores, tasks can also execute in parallel.

Our inventory bug doesn’t require parallel execution. Switching between tasks at the wrong point is enough to cause it.

What is a race condition?

A race condition happens when correctness depends on the timing or ordering of operations that can overlap.

Suppose our purchase logic looks like this:

read stock
if stock > 0:
    calculate remaining stock
    save remaining stock
    create order

Two requests could run in this order:

Request A reads stock = 1
Request B reads stock = 1
Request A writes stock = 0
Request B writes stock = 0
Both requests create an order

The final stock is zero, but we have sold two items. One request overwrote the effect of the other; this is a lost update.

Reading the stock and updating it are separate operations. Another request can run between them.

Critical sections and mutexes

The part of the code that accesses shared state and needs coordination is called a critical section. Here, it includes checking availability and reserving the item.

A mutex provides mutual exclusion: only one thread can hold that mutex at a time. Every thread accessing the protected state must follow the same locking rule.

Conceptually:

acquire inventory lock
try:
    check stock
    reserve item if available
finally:
    release inventory lock

Request B waits while A reserves the item. After A releases the lock, B checks the updated stock.

Keep the protected work short. A slow network call inside the critical section makes every waiting request wait longer.

What if threads keep waiting for each other?

Suppose an operation needs to lock two products:

Thread A holds product 1, waits for product 2
Thread B holds product 2, waits for product 1

Neither can continue. This is a deadlock.

A common way to prevent this cycle is to acquire locks in a consistent order, for example, always lock the lower product ID first. Both operations then try to acquire product 1 before product 2.

Database transactions can deadlock too. Applications need to handle a transaction being aborted when the database detects one.

Semaphores

Sometimes we want to allow several tasks to use a resource at once, up to a limit.

A semaphore maintains a count of permits. A task acquires a permit before entering and releases it after finishing.

For example, with 10 permits:

25 tasks arrive
10 can enter
15 wait for permits

This is useful for limiting access to resources such as a connection pool. It doesn’t by itself make inventory updates safe; multiple permitted tasks can still race over the same item. Semaphore documentation

Atomic operations and compare-and-swap

An atomic operation behaves as one indivisible operation to other participants. They cannot observe or interfere with a halfway-completed update.

Compare-and-swap, also called compare-and-set, is a conditional atomic update:

change the value to NEW
only if its current value is EXPECTED

If another task changes the value first, the operation fails. Our code can then re-read the state and decide what to do.

For an in-memory counter, an atomic increment can avoid a separate read, addition, and write. Protecting a larger operation involving several values requires additional coordination. Atomic variables and compare-and-set

Thread pools

Creating a new operating-system thread for every task can become expensive as traffic grows.

A fixed thread pool reuses a limited set of worker threads:

Incoming tasks -> Queue -> Worker threads

When all workers are busy, additional tasks wait in the queue. The queue also needs a capacity or overload policy, otherwise waiting work can keep growing.

A pool controls how work is executed. We still need to protect shared data accessed by its workers. Thread pools

Event loops and async/await

A backend often waits for a database or another service. An event loop lets it make progress on other work while asynchronous I/O is pending.

Node.js runs JavaScript callbacks on an event-loop thread by default, with the operating system and other runtime threads handling supporting work. A long synchronous computation can block that event loop. Node.js event loop

One JavaScript thread doesn’t remove every race condition. For example:

// Illustrative: this check and update are separate operations.
const stock = await getStock(productId);
if (stock > 0) {
  await saveStock(productId, stock - 1);
}

While one request is waiting, another can read the same stock. The purchase still needs coordination where the inventory is stored.

Producer-consumer queues

Another approach is to put purchase requests into a queue:

Requests -> Queue -> Inventory worker

The requests produce work, and the worker consumes it.

If all reservations for a product go through one worker and are processed serially, they won’t overlap at that worker. Separate products can be assigned to different workers.

A queue alone doesn’t guarantee this. Adding multiple consumers that update the same inventory can bring back the race. Delivery retries also need duplicate handling.

The API might return an order ID with a processing status, then let the user check the result later.

What changes when we add more servers?

So far, our mutex has lived inside one backend process. Now suppose we run three copies:

-> Server 1 --
Load balancer -> Server 2 ----> Shared database
             -> Server 3 --

Each process has its own in-memory lock. A lock held by Server 1 doesn’t stop Server 2 from changing the same database row.

The coordination must cover every writer. For inventory stored in one database, the database is often the right place to enforce the rule.

Pessimistic locking

With pessimistic locking, we acquire a lock before making a decision based on the row.

For example, in PostgreSQL:

BEGIN;
SELECT stock
FROM products
WHERE id = 123
FOR UPDATE;
-- Application checks the returned stock.
-- If stock > 0, reserve one item and create the order
-- within this transaction. Otherwise, make no reservation.
COMMIT;

While this transaction holds the row lock, another transaction trying to lock or update that row must wait. Ordinary snapshot reads can still proceed.

Keep the transaction short, and make sure competing purchase paths follow the same inventory rule.

Optimistic locking

With optimistic locking, we read the data and check whether it changed before accepting our update.

Suppose a product has:

stock = 1
version = 10

Both requests read version 10. Each attempts:

UPDATE products
SET stock = stock - 1,
    version = version + 1
WHERE id = 123
  AND version = 10
  AND stock > 0;

The first successful update changes the version to 11. The other request’s version check no longer matches, so it updates zero rows. Our code must check that result before creating an order.

We haven’t held an application-requested row lock throughout the read-and-decide phase. The database still takes the locks needed to execute the update.

For this simple stock rule, a conditional update can also work without a version column:

UPDATE products
SET stock = stock - 1
WHERE id = 123 AND stock > 0
RETURNING stock;

In PostgreSQL’s default Read Committed isolation, a competing update waits if needed and rechecks the condition against the updated row. No returned row means this statement reserved nothing. Any related order creation should share the reservation transaction.

Distributed locks and fencing tokens

Sometimes the shared work extends beyond a database row. For example, only one server should run a particular scheduled job.

A distributed lock coordinates participants through a shared service. Lease-based locks expire so another worker can take over if the holder stops responding.

But consider this:

Server A gets the lock, then pauses
A's lease expires
Server B gets the lock and starts working
Server A resumes and tries to write

A may still believe it owns the lock.

A fencing token helps protect the resource from stale holders. New ownership receives a larger token, for example, A gets 21 and B gets 22. Once the resource has accepted token 22, it rejects writes carrying 21.

The protected resource must enforce this check. Generating a token without checking it on writes doesn’t provide fencing.

What happens when a request is retried?

Suppose a purchase succeeds, but its response is lost. The client retries because it doesn’t know whether the first attempt worked.

Even perfectly serialized requests could create two orders if the retry is treated as a new purchase.

Idempotency means repeating the same logical operation has the same intended effect as doing it once. An API can implement this using an idempotency key:

POST /buy-product
Idempotency-Key: purchase-abc123

The client reuses that key for retries of this purchase. A new purchase gets a new key.

The server tracks the operation and its result. A completed retry can receive the stored result instead of creating another order. Claiming the key must also be atomic, so simultaneous retries can’t both start fresh work. If a payment provider is involved, use its idempotency support for that external operation too.

These are the basics. The next things worth exploring are transaction isolation, starvation, backpressure, queue delivery guarantees, and retry policies.

For practice, build a small inventory endpoint and send concurrent requests for the last item. Check the number of successful reservations, orders created, and remaining stock. Then try a conditional update or row lock, and repeat a completed request using the same idempotency key.