Much higher per-query CPU overhead compared to pgbouncer

[frangz]

Hi team,

We're running pgdog on EC2 alongside PostgreSQL 14 as a connection pooler (replacing pgbouncer for some services). We're seeing significantly higher CPU usage per query than expected, even with query_parser = "off".

Configuration

• pgdog version: v0.1.30
• Platform: Amazon Linux 2, i4i.24xlarge (96 vCPUs), x86_64
• Connection: localhost
• Workers: tested with both 2 and 6 (same per-query cost)
• Pools: 15 database pools, transaction mode, each with user authentication
• query_parser: off
• read_write_split: not set (single local server per pool, no role assigned)

Relevant config

[general]
host = "0.0.0.0"
port = 6432
openmetrics_port = 9090
workers = 2

connect_timeout = 3000
idle_timeout = 30000
server_lifetime = 300000
client_login_timeout = 60000

log_connections = false
log_disconnections = false

query_parser = "off"

[tcp]
keepalive = true
time = 300000

Sample database definition

[[databases]]
name = "api"
host = "localhost"
port = 5432
database_name = "database"
pooler_mode = "transaction"
pool_size = 150
min_pool_size = 30

We have 15 similar pools.

Observed CPU usage

Queries/s	CPU (%)	CPU per qps
400	25%	0.063%
2,000	76%	0.038%
4,000	220%	0.055%

• CPU correlates with query throughput, not with client connection count — spikes in connected clients with stable query rates do not affect CPU.
• Changing workers from 6 to 2 made no difference to CPU at the same query rate.
• Per-thread CPU (via top -H) shows 10–20% per tokio worker thread.

Comparison with pgbouncer

On the same instance, pgbouncer handles the full production workload:

Pooler	Queries/s	CPU (%)	CPU per qps
pgbouncer	7,000	50%	0.007%
pgbouncer	30,000	200%	0.007%
pgdog	4,000	220%	0.055%

Expected vs actual

Your benchmarks show pgdog with 2 workers achieving 100k+ tps at ~75% CPU on a Ryzen 7 5800X — roughly 0.00075% CPU per query. Our observed cost is ~70x higher than that benchmark, despite using query_parser = "off" and connecting via localhost.

Impact

At our peak production traffic of 30k queries/s, linear extrapolation suggests pgdog would consume ~1,650% CPU (16+ cores). This makes it impractical as a pgbouncer replacement at our scale.

Questions

• Is there additional per-query processing that happens even with query_parser = "off"?
• Could 15 pool definitions with per-pool user authentication contribute to the overhead?
• Are there any configuration options to reduce per-query CPU cost?
• Anything we are doing wrong?

[levkk]

Are you using our docker image? If not, and this might be a silly question, but did you compile it in debug mode by any chance?

cargo build --release

[frangz]

I've compiled it in the same machine, with

cargo build --release

[levkk]

Got it. Okay, let me take a look, those numbers seem completely wrong, so there must be a bug somewhere. Will report back soon.

[levkk]

Here's what I got:

CPU Usage

2 cores, roughly at 70% each, 80-95k QPS (reads-only to remove disk I/O).

TPS

$ PGPASSWORD=pgdog pgbench -h 127.0.0.1 -p 6432 -U pgdog pgdog -t 1000000 -c 100 -P 1 -S --protocol extended
pgbench (18.2, server 18.1)
starting vacuum...end.
progress: 1.0 s, 47277.2 tps, lat 1.280 ms stddev 0.289, 0 failed
progress: 2.0 s, 78480.3 tps, lat 1.259 ms stddev 0.236, 0 failed
progress: 3.0 s, 78287.1 tps, lat 1.259 ms stddev 0.239, 0 failed
progress: 4.0 s, 74479.7 tps, lat 1.320 ms stddev 0.258, 0 failed
progress: 5.0 s, 79813.4 tps, lat 1.233 ms stddev 0.238, 0 failed
progress: 6.0 s, 79221.6 tps, lat 1.240 ms stddev 0.247, 0 failed
progress: 7.0 s, 81862.6 tps, lat 1.206 ms stddev 0.231, 0 failed
^C
$ PGPASSWORD=pgdog pgbench -h 127.0.0.1 -p 6432 -U pgdog pgdog -t 1000000 -c 100 -P 1 -S --protocol simple
pgbench (18.2, server 18.1)
starting vacuum...end.
progress: 1.0 s, 53286.3 tps, lat 1.117 ms stddev 0.210, 0 failed
progress: 2.0 s, 88066.7 tps, lat 1.117 ms stddev 0.211, 0 failed
progress: 3.0 s, 85558.0 tps, lat 1.144 ms stddev 0.224, 0 failed
progress: 4.0 s, 85702.8 tps, lat 1.136 ms stddev 0.226, 0 failed
progress: 5.0 s, 90558.1 tps, lat 1.067 ms stddev 0.211, 0 failed
progress: 6.0 s, 89406.1 tps, lat 1.092 ms stddev 0.218, 0 failed
progress: 7.0 s, 93129.9 tps, lat 1.048 ms stddev 0.204, 0 failed
^C
 PGPASSWORD=pgdog pgbench -h 127.0.0.1 -p 6432 -U pgdog pgdog -t 1000000 -c 100 -P 1 -S --protocol prepared
pgbench (18.2, server 18.1)
starting vacuum...end.
progress: 1.0 s, 52471.6 tps, lat 1.114 ms stddev 0.263, 0 failed
progress: 2.0 s, 92039.7 tps, lat 1.074 ms stddev 0.214, 0 failed
progress: 3.0 s, 89935.3 tps, lat 1.092 ms stddev 0.233, 0 failed
progress: 4.0 s, 94963.1 tps, lat 1.021 ms stddev 0.219, 0 failed
progress: 5.0 s, 96003.9 tps, lat 1.020 ms stddev 0.215, 0 failed
progress: 6.0 s, 92603.7 tps, lat 1.064 ms stddev 0.232, 0 failed
progress: 7.0 s, 94951.8 tps, lat 1.027 ms stddev 0.219, 0 failed

Config

[general]
host = "0.0.0.0"
port = 6432
openmetrics_port = 9090
workers = 2

connect_timeout = 3000
idle_timeout = 30000
server_lifetime = 300000
client_login_timeout = 60000

log_connections = false
log_disconnections = false

query_parser = "off"

[tcp]
keepalive = true
time = 300000

[[databases]]
name = "pgdog"
host = "127.0.0.1"
port = 5432
database_name = "pgdog"
pooler_mode = "transaction"
pool_size = 150
min_pool_size = 30

[[databases]]
name = "pgdog1"
host = "127.0.0.1"
port = 5432
database_name = "pgdog"
pooler_mode = "transaction"
pool_size = 150
min_pool_size = 30

[[databases]]
name = "pgdog2"
host = "127.0.0.1"
port = 5432
database_name = "pgdog"
pooler_mode = "transaction"
pool_size = 150
min_pool_size = 30

[[databases]]
name = "pgdog3"
host = "127.0.0.1"
port = 5432
database_name = "pgdog"
pooler_mode = "transaction"
pool_size = 150
min_pool_size = 30

[admin]
password = "pgdog"

Thoughts

So roughly, this looks normal to me. Not sure what pgb gets, but last time I checked it was within 10% of this. CPU usage can be higher because we're multi-threaded - there is overhead there for sure. I think if you're using it as just a simple connection pooler, maybe pgb isn't such a bad choice, but if you need more features, this 10% should be worth it. That being said, the numbers you shared seem off, so could you do me a favor and run it with a profiler? We use samply, it's really easy to use:

cargo install samply
samply record target/release/pgdog

Then run your benchmark, Ctrl-C samply and give me the json it outputs. I can use the Firefox profiler to see what's up. Or maybe screenshot it if you can, maybe we'll see some hotspots.

Thanks!

P.S. I'll be afk until Sunday, so may not respond immediately.

[frangz]

Following your suggestion, I profiled pgdog using samply and found the root cause of the high CPU usage.

Root cause: SCRAM-SHA-256 client authentication

The dominant hotspot was GFp_sha256_block_data_order_avx from the ring crate, accounting for roughly 60% of CPU time. This is the SHA-256 primitive used for SCRAM-SHA-256 authentication.

We use trust for local connections, so this was entirely client-side authentication — pgdog authenticating inbound application connections, not pgdog authenticating to PostgreSQL.

SCRAM-SHA-256 uses PBKDF2 with 4,096 iterations by design. At high query throughput with connection churn across 15 pools, the cumulative cost of SCRAM handshakes dominated CPU.

Our 2 workers maxed out on CPU at around 1.5k queries/s each.

Fix

Switching auth_type to md5 resolved the issue. CPU per query dropped from roughly 0.055% to 0.014%, bringing pgdog to approximately 1.5x pgbouncer's CPU usage rather than the original 8x. This is acceptable for our use case.

After profile

With SCRAM eliminated, a second profile shows a healthy distribution:

• 58% in pgdog::frontend::listener::Listener::listen — actual connection proxying work
• 27% in tokio worker threads park/unpark cycle — async I/O event loop overhead as threads wake to handle connection events and park again
• 12% in tokio::runtime::task::core::Core::poll — async runtime scheduling

Suggestion

It may be worth documenting that SCRAM client authentication has significant CPU cost at high throughput, and that md5 is preferable for high-traffic deployments where the security trade-off is acceptable. A warning when SCRAM is configured alongside high pool counts could save others the same investigation.

Thanks for pointing me towards samply — very useful tool.

[levkk]

This is a great find. Thank you. We'll add a warning to the docs.