Module 14: Concurrency in backend systems
Everything so far, applied to a real service: request threads, connection pools, async executors, HTTP fan-out, Kafka consumers, several pods, and shutdown. This module is about keeping a service fast under load and correct when many threads, and many machines, touch the same data.
- A Spring Boot service on Tomcat has 200 request threads and a HikariCP pool of 10 connections. 200 requests that each need the database arrive at once. What happens?
Reveal
Ten run their queries; 190 threads wait inside
getConnection(). If a connection does not free up withinconnectionTimeout(30 s by default), they fail. The connection pool, not the thread count, is the real limit. - Does a
synchronizedblock stop two pods from processing the same payment twice?Reveal
No. Each pod is a separate JVM with its own memory and its own locks. Cross-pod safety needs the database (conditional updates, row locks, unique constraints) or another shared system.
- A Spring
@Servicebean has a fieldprivate Order current;set at the start of each request. What is wrong?Reveal
Beans are singletons by default, so every request thread shares that one field. Requests overwrite each other's order. Keep per-request data in local variables or parameters.
- A Kafka topic has 6 partitions. You set the Spring Kafka listener concurrency to 10. How many threads receive messages?
Reveal
Six. Each partition is assigned to at most one consumer in the group, so four consumers sit idle.
- Why can retries make an outage worse?
Reveal
Every failed call is repeated, multiplying the load on a dependency that is already struggling. Retries need limits, exponential back-off with jitter, and usually a circuit breaker.
14.1The thread model of a Spring Boot service
List the thread pools a typical Spring Boot service with a REST API, PostgreSQL, a Kafka listener and one @Scheduled job runs, and what limits each.
Reveal the answer
Web server request threads (Tomcat, 200 by default); the HikariCP connection pool (10 connections by default); Kafka listener consumer threads (one per configured concurrency, useful up to the partition count); the scheduler thread (1 by default); plus any @Async or custom executors. Each is a separate limit and they interact.
| Pool | Default | Property |
|---|---|---|
| Tomcat request threads | 200 max, 10 min spare | server.tomcat.threads.max |
| Tomcat connections | up to 8,192 open; requests waiting for a free thread queue inside Tomcat's executor (unbounded by default); the OS backlog of 100 is used only after the connection limit | server.tomcat.max-connections, server.tomcat.accept-count |
| HikariCP connections | 10 | spring.datasource.hikari.maximum-pool-size |
| HikariCP wait for a connection | 30 s | spring.datasource.hikari.connection-timeout |
@Async executor (applicationTaskExecutor) |
8 core threads, unbounded queue, so it never grows past 8 | spring.task.execution.pool.* |
@Scheduled scheduler |
1 thread | spring.task.scheduling.pool.size |
| Kafka listener threads | 1 per container | concurrency on the listener or container factory |
With spring.threads.virtual.enabled=true (Java 21, Boot 3.2+), request handling, @Async and scheduling use virtual threads instead, and the connection pool becomes the main limit even more clearly (Module 13).
14.2Thread pools vs connection pools
Each request holds a database connection for 20 ms. The pool has 10 connections. What is the maximum database-bound throughput, and what happens if traffic exceeds it?
Reveal the answer
10 connections / 0.02 s = 500 requests per second. Beyond that, requests queue for a connection; latency rises, and once waits exceed connectionTimeout they fail. Adding request threads does not help; shorter transactions or more (well-sized) connections do.
Guidelines that hold up in practice:
- Size the connection pool for the database, not for the threads. HikariCP's own guide starts from
connections = cores x 2 + effective diskson the database server, often a few dozen at most, and shared across all your pods: 8 pods x 20 = 160 connections hitting one PostgreSQL. - Keep transactions short. Never make an HTTP call or wait on a queue while holding a connection inside
@Transactional. - Fail fast. A 30 s
connectionTimeoutmeans a request thread may hang for 30 s. Many services use 1 to 5 s so overload turns into quick, visible errors. - Never nest connection use in one request (Module 11): a pool of 10 deadlocks with 10 requests that each need 2.
14.3Protecting downstream services
A payment service calls a fraud API (normally 50 ms) and a notifications API from the same 50-thread pool. The notifications API starts taking 30 s per call. What happens to payments?
Reveal the answer
Notification calls pile up and hold threads for 30 s each, soon occupying all 50, so payment requests that need a thread to call the fraud API wait or fail too. One slow, unimportant dependency takes down the important path. Fix: timeouts, and separate limits (bulkheads) per dependency.
Four tools, used together:
- Timeouts on every remote call: a connect timeout and a read (response) timeout on the client, plus an overall deadline for the request. No timeout means a thread can wait forever.
- Bulkheads: a separate bounded pool or
Semaphoreper dependency, so one slow dependency can only exhaust its own share. - Circuit breakers: after repeated failures, stop calling for a while and fail fast (or use a fallback), giving the dependency room to recover. Resilience4j provides
CircuitBreaker,Bulkhead,TimeLimiterandRetryfor Spring. - Careful retries: only for idempotent operations, a small maximum, exponential back-off with random jitter, and never stacked at every layer (3 attempts per layer x 3 layers = 27 calls for one user request).
import java.net.URI;
import java.net.http.*;
import java.time.Duration;
class FraudClient {
private final HttpClient http = HttpClient.newBuilder()
.connectTimeout(Duration.ofMillis(300)) // TCP connect limit
.build();
HttpResponse<String> check(String paymentJson) throws Exception {
HttpRequest req = HttpRequest.newBuilder(URI.create("https://fraud.internal/check"))
.timeout(Duration.ofMillis(800)) // until response headers arrive (JDK 26+: also the body)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(paymentJson))
.build();
return http.send(req, HttpResponse.BodyHandlers.ofString()); // throws HttpTimeoutException when late
}
}14.4Backpressure and overload
Traffic doubles for ten minutes. Option A: an unbounded queue in front of the workers. Option B: a bounded queue that rejects with HTTP 503 when full. Which one is still healthy at minute ten?
Reveal the answer
Option B. Option A accepts everything, so the queue and latency grow every second; by minute ten requests wait minutes, clients have timed out and retried (adding more load), and memory is filling. B serves what it can at normal latency and tells the rest to come back later.
Backpressure means letting a slow consumer push back on a fast producer instead of buffering without limit. In a JVM service:
| Technique | Effect |
|---|---|
| Bounded queues everywhere | Memory and waiting time stay limited |
Reject fast (AbortPolicy mapped to HTTP 429 or 503 with Retry-After) |
Clients back off; the service stays responsive |
CallerRunsPolicy |
The submitting thread slows down, which slows its producer |
| Semaphores in front of scarce resources | Fixed concurrency, with fast failure via tryAcquire(timeout) |
| Pull-based consumption (Kafka) | Consumers fetch only what they can process; lag grows in Kafka, not in your heap |
| Load shedding by priority | Drop low-value work (analytics, recommendations) before payments |
Every queue is either bounded or a memory leak waiting for a traffic spike. Decide, for each one, what happens when it is full.
14.5Shared state in Spring singletons
Why is a @Service with only private final fields pointing at other beans thread-safe, while one with a private BigDecimal runningTotal field is not?
Reveal the answer
Spring creates one instance of each singleton bean and every request thread calls it. Final references to thread-safe collaborators never change, so there is nothing to race on. A mutable field is shared mutable state touched by every request at once.
Per-request data in a field
@Service
public class StatementService {
private Customer customer; // shared by ALL requests
private final List<Txn> lines = new ArrayList<>();
public Statement build(long id) {
customer = repo.find(id);
lines.clear();
lines.addAll(txnRepo.forCustomer(id));
return render(customer, lines); // may render another user's data
}
}Stateless bean, per-request locals
@Service
@RequiredArgsConstructor
public class StatementService {
private final CustomerRepo repo; // final, thread-safe collaborators
private final TxnRepo txnRepo;
public Statement build(long id) {
Customer customer = repo.find(id); // local
List<Txn> lines = txnRepo.forCustomer(id); // local
return render(customer, lines);
}
}More Spring rules that come straight from earlier modules:
- Legitimately shared state (caches, counters, rate limiters) uses concurrent types:
ConcurrentHashMap,AtomicLong,LongAdder, or a cache library such as Caffeine. @Asyncneeds a deliberate executor. Define aThreadPoolTaskExecutorbean with a bounded queue, a thread name prefix and a rejection policy; the default has an unbounded queue.@Transactional, the security context and the MDC are thread-bound (Module 12). Work moved to another thread leaves them behind.- Prototype or request scope beans are rarely the answer; local variables are simpler.
@Bean(name = "notificationExecutor")
ThreadPoolTaskExecutor notificationExecutor() {
ThreadPoolTaskExecutor ex = new ThreadPoolTaskExecutor();
ex.setCorePoolSize(4);
ex.setMaxPoolSize(8);
ex.setQueueCapacity(500); // bounded
ex.setThreadNamePrefix("notify-");
ex.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
ex.setWaitForTasksToCompleteOnShutdown(true); // drain on shutdown...
ex.setAwaitTerminationSeconds(20); // ...but not forever
ex.setTaskDecorator(mdcCopyingDecorator()); // carry traceId (Module 12)
return ex;
}
// usage: @Async("notificationExecutor") public void sendReceipt(...)14.6Concurrent HTTP calls
An account overview needs the profile (80 ms), the last 10 transactions (150 ms) and the reward balance (120 ms, optional). Sketch a version that answers in about 150 ms and still works when the rewards service is down.
Reveal the answer
Start all three calls at once on a dedicated I/O executor, give rewards a short timeout with a default (completeOnTimeout plus exceptionally), combine with allOf or thenCombine, and set client timeouts so abandoned calls end too. The answer takes about as long as the slowest required call.
class AccountOverviewService {
private final ExecutorService io = new ThreadPoolExecutor(16, 16, 0, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(200), Thread.ofPlatform().name("overview-io-", 0).factory(),
new ThreadPoolExecutor.AbortPolicy());
Overview load(long customerId) {
CompletableFuture<Profile> profile = CompletableFuture.supplyAsync(() -> profiles.get(customerId), io);
CompletableFuture<List<Txn>> txns = CompletableFuture.supplyAsync(() -> ledger.last10(customerId), io);
CompletableFuture<Long> rewards = CompletableFuture.supplyAsync(() -> rewardsClient.balance(customerId), io)
.completeOnTimeout(0L, 200, TimeUnit.MILLISECONDS) // optional: default after 200 ms
.exceptionally(ex -> 0L); // optional: default on failure
return CompletableFuture.allOf(profile, txns, rewards)
.thenApply(v -> new Overview(profile.join(), txns.join(), rewards.join()))
.orTimeout(1, TimeUnit.SECONDS) // overall deadline for required parts
.join(); // one blocking wait, at the edge
}
interface Profiles { Profile get(long id); }
interface Ledger { List<Txn> last10(long id); }
interface Rewards { long balance(long id); }
record Profile(String name) { }
record Txn(long amount) { }
record Overview(Profile p, List<Txn> t, long rewards) { }
Profiles profiles = id -> new Profile("x"); Ledger ledger = id -> List.of(); Rewards rewardsClient = id -> 0L;
}Each client also needs its own connect and read timeouts (lesson 14.3), so a timed-out call does not keep an overview-io thread busy. On JDK 21 to 25, Java's HttpRequest.timeout stops at the response headers, so also bound slow bodies, for example with sendAsync(...).orTimeout(...). With virtual threads you can drop the custom pool and keep the same shape.
14.7Kafka consumers, threads and ordering
To speed up a Kafka listener, a developer hands each record to a 20-thread pool and returns immediately so the next poll happens sooner. Name two things that break.
Reveal the answer
Ordering: records from the same partition (often the same account or card) now run in parallel and can be applied out of order. Offsets: they may be committed before processing finishes, so a crash loses records, or the pool's queue grows without bound. Also, a failure in the pool no longer reaches the container's error handling and retry logic.
Facts to design around:
KafkaConsumeris not thread-safe. Use each consumer from one thread only; the only method safe to call from another thread iswakeup().- One partition goes to at most one consumer in a group. Parallelism is capped by the partition count: concurrency 10 on 6 partitions leaves 4 threads idle.
- Order is guaranteed only within a partition. Producers choose the partition by key, so "all events of account 42 in order" requires the same key and one thread at a time per key.
- Slow processing triggers rebalances. If the gap between two
poll()calls exceedsmax.poll.interval.ms(5 minutes by default), the consumer is considered dead and its partitions move. Lowermax.poll.records(default 500) if batches are slow. - Spring Kafka's
concurrencycreates that many consumers, each with its own thread. With the default ack mode (BATCH) it commits offsets after the listener has processed all records returned by a poll.
When one partition's work is too slow for one thread, process in parallel across keys but in order within a key:
import java.util.*;
import java.util.concurrent.*;
public class KeyedExecutor implements AutoCloseable {
private final ExecutorService[] lanes; // each lane runs tasks one at a time, in order
public KeyedExecutor(int n) {
lanes = new ExecutorService[n];
for (int i = 0; i < n; i++) lanes[i] = Executors.newSingleThreadExecutor();
}
public Future<?> submit(String key, Runnable task) {
int lane = Math.floorMod(key.hashCode(), lanes.length); // same key -> same lane -> same order
return lanes[lane].submit(task);
}
@Override public void close() { for (ExecutorService l : lanes) l.close(); }
public static void main(String[] args) {
Map<String, List<Integer>> seen = new ConcurrentHashMap<>();
try (KeyedExecutor ex = new KeyedExecutor(4)) {
for (int i = 0; i < 1000; i++) {
String key = "acct-" + (i % 7);
int seq = i;
ex.submit(key, () -> seen.computeIfAbsent(key, k -> Collections.synchronizedList(new ArrayList<>())).add(seq));
}
}
boolean ordered = seen.values().stream().allMatch(l -> {
for (int i = 1; i < l.size(); i++) if (l.get(i) < l.get(i - 1)) return false;
return true;
});
System.out.println("7 keys on 4 lanes, every key in order: " + ordered);
}
}With hand-off like this, commit an offset only after every record up to it has finished. That gives at-least-once delivery (a crash replays unfinished records), so handlers must be idempotent; committing earlier can lose records on a crash. The simplest correct design is still: more partitions, more consumers, processing inside the listener thread.
concurrency = 3 on 6 partitions: each consumer thread owns two partitions. More consumers than partitions sit idle.KeyedExecutor above implements it.Idempotent consumer: with at-least-once delivery, the same event can arrive twice (after a rebalance or a retry). Record each event ID in the same transaction as its effect:
@KafkaListener(topics = "payouts", concurrency = "3")
@Transactional
public void onPayout(PayoutEvent e) {
int inserted = jdbc.update(
"INSERT INTO processed_event (event_id) VALUES (?) ON CONFLICT DO NOTHING", e.id());
if (inserted == 0) return; // already handled: a redelivery, do nothing
payoutService.process(e); // same transaction as the dedupe row
}14.8Many pods: JVM locks stop at the process boundary
Your payout service runs on 4 pods. The code wraps "check payout not yet sent, then send it and mark sent" in synchronized. A retry from the client lands on a different pod at the same moment. What happens?
Reveal the answer
Both pods pass the check, because each synchronized only excludes threads inside its own JVM. The payout is sent twice. The check and the mark must be atomic in the shared database (or another shared system), not in memory.
Ways to make it safe across pods, simplest first:
| Technique | How | Good for |
|---|---|---|
| Conditional update | UPDATE payout SET status = 'SENT' WHERE id = ? AND status = 'PENDING' and check that 1 row changed |
State transitions, balance checks |
| Unique constraint / idempotency key | Insert a row keyed by the request's idempotency key; a duplicate insert fails | Exactly-once effects from retried requests and redelivered messages |
| Optimistic locking | A version column; UPDATE ... SET ..., version = version + 1 WHERE id = ? AND version = ? (JPA @Version does this) and retry on conflict |
Low-contention edits |
| Pessimistic row lock | SELECT ... FOR UPDATE inside a transaction, then update |
High-contention rows where retries would thrash |
| Distributed lock (Redis, ZooKeeper, a DB lock table) | Acquire a lease before the work | Coordinating work that is not a single row; needs lease expiry and fencing tokens, because a paused holder may still act after its lease expires |
-- Exactly one pod wins this transition; the others see 0 rows updated and stop.
UPDATE payout
SET status = 'SENT', sent_at = now()
WHERE id = :id
AND status = 'PENDING';
-- Idempotency: a retried request with the same key cannot create a second payment.
INSERT INTO payment_request (idempotency_key, account_id, amount_paise)
VALUES (:key, :account, :amount)
ON CONFLICT (idempotency_key) DO NOTHING;Database-level idempotency beats an in-memory or cache-based dedupe for anything financial: the database survives restarts, sees every pod, and enforces the rule in the same transaction as the write. A Redis key with a TTL can expire or be lost before the duplicate arrives.
Optimistic locking in JPA: a @Version column makes every update conditional on the version you read; a concurrent change makes your save fail instead of overwriting it.
@Entity
class CreditLimit {
@Id Long accountId;
long limitPaise;
@Version long version; // JPA adds "WHERE version = ?" and increments it on every update
}
// In a service: retry a few times when another pod updated the row first
for (int attempt = 1; ; attempt++) {
try {
txTemplate.executeWithoutResult(s -> {
CreditLimit cl = repo.findById(id).orElseThrow();
cl.limitPaise += increasePaise; // saved at commit with the version check
});
break;
} catch (ObjectOptimisticLockingFailureException e) {
if (attempt == 3) throw e; // give up after 3 tries
}
}14.9Graceful shutdown
Kubernetes sends SIGTERM to a pod during a deploy and kills it 30 seconds later. List what the service should do in those 30 seconds.
Reveal the answer
Stop receiving new work (fail the readiness probe, stop accepting connections, stop polling Kafka), let in-flight requests and tasks finish, commit Kafka offsets for completed records, shut executors down with a timeout (interrupting stragglers), close connection pools, and exit before the deadline.
finally blocks.server:
shutdown: graceful # default since Spring Boot 3.4; stop accepting, finish in-flight
spring:
lifecycle:
timeout-per-shutdown-phase: 20s # default 30s; keep the total below the pod's grace period- Executors you create yourself need their own shutdown: a
@PreDestroymethod withshutdown(),awaitTermination(timeout)andshutdownNow()(Module 6), or Spring'sThreadPoolTaskExecutorsettings shown in lesson 14.5. - Tasks must respond to interruption (Module 5) or
shutdownNow()cannot stop them. - Kafka listener containers stop polling, let the current records finish and commit offsets when the context closes; very long processing still needs to fit inside the timeout.
- Make work idempotent, because a hard kill can always happen (OOM, node failure). Graceful shutdown reduces duplicates; it cannot eliminate them.
14.10Use cases: the service problem and the tool
Your payment API's p99 latency jumped from 200 ms to 30 s during a partner outage, although the partner is only used for SMS receipts. Which two missing protections explain it?
Reveal the answer
No timeout on the SMS client (calls hang for the partner's full delay) and no bulkhead (those hanging calls used the same threads as payments). With a short timeout and a small separate limit for SMS, payments would have stayed fast.
Throughput = connections / transaction time.
Lesson 14.3 and Lab 8.
Backpressure, lesson 14.4.
Per-request data in locals.
Lesson 14.6 and Lab 5.
Lesson 14.7.
Lesson 14.8.
Lesson 14.9.
14.11Common mistakes
Common mistakes
- More request threads to fix a slow database. The connection pool and the database are the limit.
- HTTP calls inside
@Transactional, holding a connection for the whole remote call. - No timeouts on remote calls, so one slow dependency holds threads forever.
- One shared pool for every dependency, with no bulkheads.
- Retries without back-off, jitter or limits, at several layers.
- Mutable per-request fields in singleton beans.
- Handing Kafka records to a pool and losing ordering and offset safety.
synchronizedas protection against duplicate processing across pods.- Ignoring shutdown: pools that never drain, tasks that ignore interrupts, work that is not idempotent.
14.12Self-check
- What limits database throughput more: request threads or connections?
Reveal
Connections: throughput is at most connections divided by time per transaction; extra threads just wait for a connection.
- What is a bulkhead?
Reveal
A separate limit (pool or semaphore) per dependency, so one slow dependency can only exhaust its own share of threads.
- Why is the default
@Asyncexecutor risky?Reveal
With platform threads it has an unbounded queue, so it never grows past its 8 core threads and tasks pile up in memory under load. With
spring.threads.virtual.enabled=trueit has no queue and no concurrency limit at all unless you setspring.task.execution.simple.concurrency-limit. - How do you process one Kafka partition faster while keeping per-account order?
Reveal
Hash the key to one of N single-threaded lanes so different keys run in parallel and each key stays sequential, and commit offsets only after processing completes (or make handlers idempotent). More partitions is the simpler first option.
- Name two database techniques that prevent double processing across pods.
Reveal
Any two of: a conditional
UPDATE ... WHERE status = 'PENDING'checking the row count, a unique constraint on an idempotency key, optimistic locking with a version column,SELECT ... FOR UPDATE.
Key takeaways
- Know your service's thread pools and their defaults; the connection pool is usually the real limit.
- Every remote call gets a timeout; every dependency gets its own limit (bulkhead).
- Bound every queue and decide what happens when it is full.
- Singleton beans must be stateless or use concurrent types; context stays on its thread.
- Kafka: each partition is read by at most one consumer thread in the group, order holds only within a partition, commit after processing, handlers idempotent.
- Across pods, only shared state (the database) prevents duplicates; shut down gracefully within the grace period.
Mastery checklist
Tick an item only when it is true with your notes closed.
Module 15: Debugging and testing concurrent code
Concurrency bugs hide in tests and appear under load. This module covers taking and reading thread dumps, diagnosing deadlocks, contention, pool exhaustion and stuck I/O, and writing tests that actually make threads overlap, plus the tools for going deeper.
- A service is stuck, CPU near zero. What is the first thing to capture before restarting it?
Reveal
Two or three thread dumps a few seconds apart (
jcmd <pid> Thread.print). A restart destroys the evidence. - In a dump, 40 threads show
BLOCKED (on object monitor)with- waiting to lock <0x...a2e0>. How do you find the culprit?Reveal
Search the dump for
- locked <0x...a2e0>. The thread holding that monitor is the one everyone waits for; its stack shows what it is doing while holding the lock. - All Tomcat threads show
TIMED_WAITING (parking)insideHikariPool.getConnection. What is wrong?Reveal
Connection pool exhaustion: every request waits for a database connection. Look for slow queries, long transactions, leaked connections, or a pool far too small for the load.
- A race test passes 100 times in a row. Does that prove the code is thread-safe?
Reveal
No. A passing test only shows the race did not happen in those runs. A failing run proves a bug; passing runs prove nothing about its absence.
- Why is
Thread.sleep(500)a poor way to wait for async work in a test?Reveal
It is either too long (slow tests) or too short on a loaded CI machine (flaky tests). Wait for the actual event with a latch, a future or Awaitility, with a timeout.
15.1Taking a thread dump
A pod in Kubernetes is unresponsive. You can exec into it. Which command produces a thread dump, and why take more than one?
Reveal the answer
kubectl exec <pod> -- jcmd 1 Thread.print > dump1.txt (the JVM is often PID 1 in a container), then again 5 to 10 seconds later. One dump is a snapshot; comparing several shows which threads are truly stuck (same stack every time) and which are just busy.
A thread dump lists every thread in the JVM with its name, state and current stack, plus the locks it holds and waits for.
| Command | Notes |
|---|---|
jcmd <pid> Thread.print |
The recommended tool; add -l for java.util.concurrent lock details |
jstack -l <pid> |
Classic; -l adds "Locked ownable synchronizers" (ReentrantLock and friends) |
kill -3 <pid> |
The JVM prints the dump to its own standard output (your container logs) |
jcmd <pid> Thread.dump_to_file -format=json dump.json |
Includes virtual threads, which Thread.print does not list |
| JDK Mission Control, VisualVM, IntelliJ | Graphical; same data |
Spring Boot Actuator /actuator/threaddump |
Over HTTP, if the endpoint is exposed (protect it) |
Find the PID with jcmd (no arguments) or jps. Take three dumps 5 to 10 seconds apart, and note the time and what the service was doing (CPU, latency, error rate).
Example: a dump from inside the JVM, for a watchdog that logs one automatically when latency spikes:
import java.lang.management.*;
public class SelfDump {
public static void main(String[] args) {
ThreadMXBean mx = ManagementFactory.getThreadMXBean();
for (ThreadInfo t : mx.dumpAllThreads(true, true)) { // include monitors and j.u.c locks held
System.out.println(t.getThreadName() + " " + t.getThreadState());
}
}
}ThreadInfo.toString() prints a short stack too. Rate-limit automatic dumps; a dump of thousands of threads is large.
15.2Reading a thread dump
A thread's entry says java.lang.Thread.State: TIMED_WAITING (sleeping) and a few lines below - locked <0x00000000a851a2e0>. Why is that worth a second look?
Reveal the answer
It is sleeping while holding a lock. Every other thread that needs that lock is BLOCKED until the sleep ends. Sleeping, I/O or remote calls inside a lock (Module 3) turn into exactly this picture.
"ledger-writer-1" #20 prio=5 os_prio=0 cpu=0.11ms elapsed=3.37s tid=0x00007f251813bb50 nid=3526 waiting on condition
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@21/Thread.java:509)
at Dump.sleep(Dump.java:6)
at Dump.writeLedger(Dump.java:7)
- locked <0x00000000a851a2e0> (a java.lang.Object) <-- holds the lock while sleeping
"ledger-writer-2" #21 prio=5 os_prio=0 cpu=0.35ms elapsed=3.37s tid=0x00007f251813ccc0 nid=3527 waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
at Dump.writeLedger(Dump.java:7)
- waiting to lock <0x00000000a851a2e0> (a java.lang.Object) <-- same address: waits for writer-1
"payout-worker-1" #23 prio=5 os_prio=0 cpu=0.71ms elapsed=3.36s tid=0x00007f251815c7b0 nid=3529 waiting on condition
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
- parking to wait for <0x00000000a85d4518> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@21/LinkedBlockingQueue.java:435)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21/ThreadPoolExecutor.java:1070)
<-- an idle pool worker: normal
"http-nio-8080-exec-1" #24 prio=5 os_prio=0 cpu=0.65ms elapsed=3.36s tid=0x00007f25181864a0 nid=3530 waiting on condition
java.lang.Thread.State: WAITING (parking)
- parking to wait for <0x00000000a843f930> (a java.util.concurrent.FutureTask)
at java.util.concurrent.FutureTask.get(java.base@21/FutureTask.java:190)
at Dump.lambda$main$5(Dump.java:16) <-- a request thread blocked on a FutureHow to read each part:
| Part | Meaning |
|---|---|
"ledger-writer-1" #20 |
Thread name and Java thread number. Good names make dumps readable |
nid=3526 |
The OS thread ID; match it with top -H -p <pid> to find threads burning CPU |
State: BLOCKED (on object monitor) |
Waiting to enter synchronized |
State: WAITING (parking) |
Parked by LockSupport: ReentrantLock, queues, futures, latches |
State: TIMED_WAITING (sleeping) |
Inside Thread.sleep |
State: WAITING (on object monitor) |
Inside Object.wait() |
- locked <addr> |
This thread holds that monitor |
- waiting to lock <addr> |
This thread wants that monitor |
- parking to wait for <addr> (a ...) |
What it is parked on: the class name tells you a lock, a queue condition or a FutureTask |
Locked ownable synchronizers |
ReentrantLocks and similar held by the thread (shown with -l) |
Most of a healthy dump is idle threads: pool workers parked in getTask / take, the JVM's own GC and compiler threads. Look for the unusual: many threads in the same frame, threads holding locks while waiting, the same stack in every dump.
15.3Diagnosing from dumps
Across three dumps 10 seconds apart, one thread shows RUNNABLE at the same line of parseStatement() every time, and the CPU is pinned at 100% of one core. What is likely wrong?
Reveal the answer
A hot or infinite loop in parseStatement() (or very heavy work on one huge input). The thread is genuinely running, not waiting, and never leaves that code. Map its nid to the CPU usage with top -H to confirm.
| Pattern in the dumps | Likely cause | Next step |
|---|---|---|
Found one Java-level deadlock |
Lock-ordering deadlock | Read the two stacks; fix the lock order or remove a lock |
Many threads BLOCKED on one address |
Lock contention; the holder is slow | Find - locked <addr>; move I/O out of the lock, split the lock |
Request threads TIMED_WAITING (parking) in HikariPool.getConnection (ConcurrentBag.borrow) |
Connection pool exhausted | Check slow queries, long transactions, leaked connections, pool size |
Request threads WAITING in FutureTask.get or CompletableFuture.join |
Waiting on a saturated executor or a slow dependency | Check that pool's threads; add timeouts; bulkheads |
Threads RUNNABLE in socketRead / Net.poll in every dump |
Remote call with no (or a huge) read timeout | Add client timeouts |
RUNNABLE in the same frames, high CPU |
Hot loop, regex backtracking, huge input | Profile (async-profiler, JFR) |
Pool threads all RUNNABLE in your code, queue growing |
Pool too small or tasks too slow for the arrival rate | Little's law, sizing, backpressure (Module 6) |
Add executor metrics before you need them: active threads, queue size, completed tasks and task duration for every pool, plus HikariCP's active, idle and pending connections. Micrometer's ExecutorServiceMetrics and Hikari integration cover most of these (task duration needs the executor wrapped with ExecutorServiceMetrics.monitor); count rejections yourself in the RejectedExecutionHandler. A dashboard often answers the question before anyone opens a dump.
15.4Testing for race conditions
Your first race test starts two threads that each call increment() 1,000 times and checks the total. It never fails, even on code you know is broken. Why, and how do you make it fail?
Reveal the answer
The first thread often finishes before the second even starts, so they never overlap. Make them overlap: many threads, a start gate (CountDownLatch(1)) so all begin at the same moment, more iterations, and repeat the whole test many times.
Make races likely, then check invariants:
- Many threads, more than cores, so the scheduler interleaves them.
- A start gate: every thread waits on one
CountDownLatch, then all are released together. - Enough iterations per thread to overlap for a while, and repeat the test (
@RepeatedTest). - Assert an invariant at the end: the total, no duplicate IDs, balance equals the sum of entries.
- Use timeouts on every wait, so a deadlock fails the test instead of hanging CI.
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
public class RaceHarness {
interface Counter { void increment(); int get(); }
static class Unsafe implements Counter { int n; public void increment() { n++; } public int get() { return n; } }
static class Safe implements Counter { final AtomicInteger n = new AtomicInteger(); public void increment() { n.incrementAndGet(); } public int get() { return n.get(); } }
static boolean runOnce(Counter c, int threads, int perThread) throws InterruptedException {
ExecutorService pool = Executors.newFixedThreadPool(threads);
CountDownLatch startGate = new CountDownLatch(1);
CountDownLatch done = new CountDownLatch(threads);
for (int t = 0; t < threads; t++) {
pool.execute(() -> {
try {
startGate.await(); // everyone waits here...
for (int i = 0; i < perThread; i++) c.increment();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
done.countDown();
}
});
}
startGate.countDown(); // ...and starts at the same instant
boolean finished = done.await(10, TimeUnit.SECONDS); // never hang the build
pool.shutdownNow();
return finished && c.get() == threads * perThread; // the invariant
}
public static void main(String[] args) throws InterruptedException {
int unsafeFails = 0, safeOk = 0, runs = 20;
for (int r = 0; r < runs; r++) {
if (!runOnce(new Unsafe(), 8, 10_000)) unsafeFails++;
if (runOnce(new Safe(), 8, 10_000)) safeOk++;
}
System.out.println("unsafe counter failed " + unsafeFails + " of " + runs + " runs");
System.out.println("safe counter correct " + safeOk + " of " + runs + " runs");
}
}The same structure as a JUnit 5 test:
@RepeatedTest(20)
@Timeout(10)
void walletNeverOverdraws() throws Exception {
Wallet wallet = new Wallet(1_000); // balance in paise
int threads = 16;
ExecutorService pool = Executors.newFixedThreadPool(threads);
CountDownLatch start = new CountDownLatch(1);
List<Future<Boolean>> results = new ArrayList<>();
for (int i = 0; i < threads; i++) {
results.add(pool.submit(() -> { start.await(); return wallet.debit(100, "t"); }));
}
start.countDown();
long approved = 0;
for (Future<Boolean> f : results) if (f.get(5, TimeUnit.SECONDS)) approved++;
pool.shutdown();
assertEquals(10, approved); // exactly 1000 / 100 debits succeed
assertEquals(0, wallet.balance());
}15.5Testing async code
A service sends a receipt on an executor after a payment. How can a unit test check that the receipt was sent, without Thread.sleep and without flakiness?
Reveal the answer
Inject the executor. In the unit test pass a direct executor (Runnable::run), so the task runs immediately on the test thread and the assertion follows it deterministically. In an integration test with a real pool, wait for the effect with a latch, a returned future or Awaitility, always with a timeout.
class ReceiptService {
private final Executor executor; // injected, not created inside
private final Mailer mailer;
ReceiptService(Executor executor, Mailer mailer) { this.executor = executor; this.mailer = mailer; }
void paymentSucceeded(String paymentId) {
executor.execute(() -> mailer.send("receipt for " + paymentId));
}
interface Mailer { void send(String body); }
}class ReceiptServiceTest {
@Test
void sendsReceipt() {
List<String> sent = new ArrayList<>();
ReceiptService svc = new ReceiptService(Runnable::run, sent::add); // direct executor: runs inline
svc.paymentSucceeded("p-42");
assertEquals(List.of("receipt for p-42"), sent); // no sleep, no flakiness
}
}// Integration test with real threads: wait for the effect, never sleep blindly
await().atMost(Duration.ofSeconds(5))
.untilAsserted(() -> assertEquals(1, mailbox.count("receipt for p-42"))); // Awaitility- Code that returns a
CompletableFutureis easy to test:assertEquals(x, future.get(5, SECONDS)). - Test failure paths too: a dependency that throws, a timeout, a rejection from a full pool.
- Keep a test timeout on everything that touches threads (
@Timeoutin JUnit 5).
15.6Going deeper: jcstress, JFR and profilers
When normal tests are not enough:
| Tool | What it does | When to reach for it |
|---|---|---|
| jcstress (OpenJDK) | Runs tiny concurrent snippets millions of times across many interleavings and reports which outcomes occurred ("acceptable", "forbidden", "interesting") | Verifying memory-model questions: publication, volatile, lock-free code |
| Java Flight Recorder (JFR) | Low-overhead recording built into the JVM; events such as jdk.JavaMonitorEnter (contended synchronized), jdk.ThreadPark, jdk.VirtualThreadPinned |
Contention and pinning in production: jcmd <pid> JFR.start duration=60s filename=rec.jfr |
| JDK Mission Control | Opens JFR recordings; lock and thread views | Analysing a recording |
| async-profiler | CPU, allocation and lock profiling with flame graphs | Hot loops, contention hot spots |
| Micrometer executor and HikariCP metrics | Continuous numbers for pools | Early warning before an incident |
@JCStressTest
@Outcome(id = "1, 1", expect = Expect.ACCEPTABLE, desc = "saw both writes")
@Outcome(id = "0, 0", expect = Expect.ACCEPTABLE, desc = "saw neither")
@Outcome(id = "1, 0", expect = Expect.ACCEPTABLE_INTERESTING, desc = "saw flag but not data: reordering")
@State
public class PlainPublication {
int data; boolean ready; // try making 'ready' volatile: "1, 0" disappears
@Actor public void writer() { data = 1; ready = true; }
@Actor public void reader(II_Result r) { boolean ok = ready; r.r1 = ok ? 1 : 0; r.r2 = ok ? data : 0; }
}15.7Use cases: an incident checklist
Half an hour after a deploy, response times climb and the pod's memory keeps growing, but CPU is low. Name two likely causes and the first two things you capture.
Reveal the answer
A growing unbounded queue (requests or tasks piling up behind a slow dependency or an exhausted pool), or ThreadLocal/cache data being retained. Capture thread dumps (where are threads waiting?) and pool and queue metrics; take a heap dump if memory keeps rising.
Look for BLOCKED on one lock, parking in getConnection or Future.get, socket reads.
Map hot nids to stacks; async-profiler or JFR for the details.
A changed pool size or a missing timeout is the usual suspect.
Unbounded queues, ThreadLocal leaks, caches without limits.
In-memory or across pods; fix with atomic steps or DB constraints.
A task ignoring interrupts or an unbounded wait.
Replace sleeps with latches and futures.
15.8Common mistakes
Common mistakes
- Restarting a stuck service without taking dumps.
- Taking one dump and treating a busy thread as a stuck one.
- Race tests where threads never overlap: no start gate, too few threads or iterations.
Thread.sleepin tests to wait for async work.- No timeouts in concurrent tests, so a deadlock hangs CI instead of failing.
- Executors created inside classes, which makes them impossible to replace in tests.
- No pool metrics, so exhaustion is discovered by users first.
15.9Self-check
- Which command dumps virtual threads too?
Reveal
jcmd <pid> Thread.dump_to_file -format=json <file>. - What does
WAITING (parking)withparking to wait for <...> (a java.util.concurrent.FutureTask)mean?Reveal
The thread called
get()on a future and is waiting for that task to finish. - How do you find which thread holds a contended monitor?
Reveal
Take the address from
- waiting to lock <addr>and search the dump for- locked <addr>. - Why does a race test need a start gate?
Reveal
Without it, threads start at different times and may never run their critical sections at the same moment; the gate releases them together.
- How do you make async code deterministic in unit tests?
Reveal
Inject the
Executorand pass a direct executor (Runnable::run) so tasks run inline on the test thread.
Key takeaways
- Take two or three thread dumps a few seconds apart before restarting anything.
- Read state plus stack: BLOCKED on a monitor, parked on a lock, queue or future, RUNNABLE in I/O or code.
- Five patterns explain most incidents: deadlock, contention, pool exhaustion, missing timeouts, hot loops.
- Race tests need many threads, a start gate, repetition, an invariant and timeouts.
- Make async code testable by injecting executors; wait for effects, never sleep.
- Export pool, queue and connection metrics so dashboards warn you before users do.
Mastery checklist
Tick an item only when it is true with your notes closed.