The Importance of Load Testing — A Java 21 Loom Server's Journey from 4 to 17,000 polls/sec

Starting Point: 4 ops/sec

Secure Chats' Java backend went to AWS this spring with virtual threads on. One thread per request, no servlet pool to size, no read timeouts to babysit. Project Loom delivered what it promised. The first load test, against the same server running locally with 500 simulated polling clients, opened with this:

==> Running POLL workload for 120s
[t=   5s]  issued=527  ok=27   err=0  window:  105.0 ops/sec  inflight=500
[t=  10s]  issued=527  ok=27   err=0  window:    0.0 ops/sec  inflight=500
[t=  15s]  issued=527  ok=27   err=0  window:    0.0 ops/sec  inflight=500
...
[t= 120s]  issued=527  ok=27   err=0  window:    0.0 ops/sec  inflight=500

Twenty-seven successful polls out of five hundred, then 105 seconds of complete silence. inflight=500 the entire run. The server was alive — it had parked 500 connections successfully — but its throughput was effectively zero.

The number 27 turned out to be the first hint. It's roughly the number of carrier threads the JVM uses by default for the virtual-thread scheduler. Without knowing it, we'd built a server that scaled with CPU cores, not with virtual threads.

The Diagnosis: synchronized Pins Virtual Threads

Loom's headline contract is "blocking is free again". A virtual thread that calls blocking I/O is unmounted from its carrier; the carrier runs another virtual thread. One operating-system thread can serve hundreds of thousands of virtual ones.

The fine print in JEP 444 (Virtual Threads, Java 21 GA): a virtual thread inside a synchronized block or method is pinned to its carrier until it exits the block. While pinned, the carrier cannot run other virtual threads. If that virtual thread then calls a blocking operation — Object.wait(), InputStream.read(), Thread.sleep() — the carrier sits idle holding it.

A scheduler default of ~CPU-count carriers, on a server whose request path runs through several synchronized blocks, collapses to "max N cores concurrent active requests" no matter how many virtual threads exist. Our 27 was eight cores' worth of carriers spinning through a tight rotation while 473 server-side VTs queued waiting for a carrier slot.

The fix is well-known: use java.util.concurrent.locks primitives. They use LockSupport.park, which Loom understands — a virtual thread blocked on a ReentrantLock is unmounted from its carrier just like a virtual thread blocked on a socket.

What surprised us was how many synchronized blocks were on the request path, and how non-obvious some of them were. Five rounds of fixes later, the same test landed at 17,000 ops/sec.

Fix 1 — Stop Printing on the Hot Path

Two leftover debug printlns in the Java client were dumping the raw response body and every outgoing message to System.err / System.out:

// Client.java — getNotifications, line 145
System.err.println(">>> RAW response body: " + response.body());

// Client.java — postMessage, line 310
System.out.println("Sending " + sendMsg);

PrintStream.println is synchronized, and a virtual thread inside a synchronized block whose body blocks on terminal I/O is the perfect contention recipe. When the macOS terminal hits buffer pressure, every println on every virtual thread waits behind it. The carriers can't unmount — they're pinned inside the synchronized region.

The fix here was just to delete the lines. They had no business being on the request hot path on any Java version. We also added a one-time System.setErr(fileStream) redirect in the load test driver so any future stray println in the production client jar lands in a file instead of the terminal — defence in depth.

Fix 2 — Async the Logger

The server's logger has two paths. With log_file_dir set in config, Log.info hands the message to an ArrayBlockingQueue and returns; a single drain thread does the file I/O off the request path. Without log_file_dir set, every Log.info falls through to System.out.println(...) on the calling thread.

The shipped config didn't set log_file_dir. Every server request was running the synchronous fallback. PrintStream is synchronized; Java 21 pins; the same contention story as Fix 1 reappeared, this time on the server side, with all 500 server-side virtual threads serializing on System.out's monitor.

The fix was a one-line addition to the config that the load-test runner now writes automatically. After this, the per-second log file showed 23,915 lines for a 15-second window — the drain thread was eating everything the request path produced, off the hot path, with the request path no longer blocking on file I/O.

Fix 3 — Replace the Homegrown ReadWriteLock

With logging cleared, the next pinning trace pointed at the auth path. Every authenticated request — which is every request except /register — calls OnDiskDS.authenticate(user), which calls getPrincipalFromMemory (read lock) followed by storePrincipalInMemory (write lock), twice. The read-write lock was a vintage homegrown implementation built on synchronized methods plus Object.wait() for blocking. Every method on it was a pin site:

public synchronized void acquireRead() throws InterruptedException {
    ...
    while (currentWritingThread != null || mWaitingWriters > 0) {
        wait();   // <-- pinned inside synchronized method
    }
    ...
}

The fix was a thin wrapper around java.util.concurrent.locks.ReentrantReadWriteLock, preserving the existing API so the callers didn't need to change:

public class ReadWriteLock {
    private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();

    public void acquireRead() throws InterruptedException {
        lock.readLock().lockInterruptibly();
    }
    public void releaseRead() {
        try { lock.readLock().unlock(); }
        catch (IllegalMonitorStateException ignored) {} // preserve old tolerant behaviour
    }
    public void acquireWrite() throws InterruptedException {
        lock.writeLock().lockInterruptibly();
    }
    public void releaseWrite() {
        try { lock.writeLock().unlock(); }
        catch (IllegalMonitorStateException ignored) {}
    }
}

Re-running the 500-user test after this single change took us from 4 ops/sec to 822 ops/sec — a 200× improvement from removing one lock implementation.

Fix 4 — Rate Limiter

At 2000 users the pinning trace pointed at CachedData.isRateLimited line 57: a synchronized (existing) block guarding a per-user counter. Conceptually uncontended — each user has its own TrafficInfo — but every entry into a synchronized block on Java 21 still consumes a carrier slot. With 200 carriers and 2000 virtual threads all wanting to do their brief synchronized check, 1800 were queued waiting for a carrier at any moment.

Same one-line transformation: synchronized (existing) { ... } → existing.lock.lock(); try { ... } finally { existing.lock.unlock(); }, with a ReentrantLock added to TrafficInfo. The CachedData pinning trace disappeared from the server log on the next run. Throughput at 5000 users jumped to 17,455 ops/sec.

Fix 5 — MessageQueue.wait()

The send path uses a different mechanism: MessageQueue.waitForMessages parks a request on Object.wait(TIMEOUT) until a matching dispatchMessage wakes it via notifyAll(). The original used synchronized + wait():

synchronized (messageBox) {
    try {
        messageBox.wait(TIMEOUT_MILLIS);  // pin for up to TIMEOUT_MILLIS
    } catch (InterruptedException ignored) {}
}

A virtual thread inside Object.wait() inside synchronized holds its carrier for the entire wait. With a default 5-minute server timeout, a single long-poll could lock a carrier for five minutes.

We swapped synchronized + Object.wait() for ReentrantLock + Condition.awaitNanos() and incidentally fixed a lost-notify race the original had — an explicit done flag is now checked before each await, and the wait loops on spurious wakeups:

messageBox.lock.lock();
try {
    long nanos = TimeUnit.MILLISECONDS.toNanos(TIMEOUT_MILLIS);
    while (!messageBox.done) {
        if (nanos <= 0L) break;             // timed out
        nanos = messageBox.messageReady.awaitNanos(nanos);
    }
} finally {
    messageBox.lock.unlock();
}

The send-mode load test at 100 sender pairs ran cleanly — 968 msg/sec, zero errors, p50 = 44 ms, no pinning traces in the server log.

The Final Numbers

The peak ceiling of ~17k ops/sec on this hardware turned out to be SQLite's populate select, not anything pinning-related. The journey took us from "server effectively dead at 500 clients" to "server comfortably handles 10,000 concurrent long-poll clients with zero errors". Same hardware, same JDK, six small code changes.

The Real Lessons: Load Test, Then Upgrade to JDK 25

Lesson 1 — The harness was the diagnosis.

Every fix here came from the load test, and none would have been found any other way. Unit tests passed, e2e tests passed, production was running — normal traffic stayed under the carrier ceiling, so the pinning was invisible. The harness paid for itself five times over: log-stream serialization, ReadWriteLock pinning, CachedData pinning, MessageQueue pinning, plus a plaintext-key debug print on the encryption path that no functional test would catch. A no-deps single-file load harness in the same repo as your e2e tests, run with -Djdk.tracePinnedThreads=short, is a startlingly high-leverage investment.

Lesson 2 — The Java version is the architecture decision.

Fixes 3, 4 and 5 are all workarounds for JEP 491 — Synchronize Virtual Threads without Pinning, preview in Java 24 and GA in Java 25 LTS (September 2025). On JDK 25, synchronized no longer pins virtual threads; the original ReadWriteLock, CachedData and MessageQueue all work correctly, and the same load test hits ~17k ops/sec against the unmodified code.

Pragmatic recommendation: on Java 25, write the natural Java — synchronized is fine, Object.wait() is fine. On Java 21, every synchronized on the request path is a potential pin point; audit, swap to java.util.concurrent.locks, run -Djdk.tracePinnedThreads=short as part of your load tests, and plan the upgrade.

Load-test harness in this post: a single-file LoadTest.java in the Secure Chats E2E directory, ~420 lines, no JUnit dependency. Server changes: ~30 lines across ReadWriteLock.java, CachedData.java, and MessageQueue.java. Reference client: github.com/Secchats-tech/ChatJavaUI.