perf(spanner): avoid redundant end() calls on completed HTTP/2 streams

[olavloite]

Summary of the Workaround

This PR introduces a targeted workaround in @google-cloud/spanner for an upstream issue in @grpc/grpc-js (grpc/grpc-node#3082) where redundant .end() calls are issued on finished HTTP/2 streams.

The Problem

1. In @grpc/grpc-js, when executing unary and server-streaming RPCs, the client marks the request half-closed (halfClose()) after sending the request payload, which sets this.http2Stream.writableEnded = true.
2. When the server completes the call and sends trailing metadata, @grpc/grpc-js invokes destroyHttp2Stream(), which calls this.http2Stream.end().
3. In Node.js stream internals (node:internal/streams/writable), calling .end() on a stream that already has writableEnded = true without providing a callback causes Node to instantiate new ERR_STREAM_ALREADY_FINISHED('end') with a full synchronous V8 stack trace capture. Because no callback was provided, the constructed error is immediately discarded.
4. Under sustained or bursty query load, creating and discarding stack traces synchronously on the Node.js main thread consumes CPU cycles and increases garbage collection pressure, leading to event-loop delays and higher tail latency.

The Implementation

Rather than modifying global Node.js stream prototypes (such as stream.Duplex.prototype), the workaround targets only @grpc/grpc-js internals:

• Targeted Patch: Patches Http2SubchannelCall.prototype.destroyHttp2Stream and Http2SubchannelCall.prototype.halfClose in @grpc/grpc-js:
  • In destroyHttp2Stream(): returns early if this.http2Stream.destroyed is already true, or if this.serverEndedCall && this.http2Stream.writableEnded is true (avoiding the second .end() call).
  • In halfClose(): returns early if this.http2Stream.destroyed or this.http2Stream.writableEnded is already true.
  • In all other cases, delegates directly to the original implementation.
• Scope: Leaves stream.Duplex.prototype, net.Socket, tls.TLSSocket, and other Node.js stream types untouched.
• Idempotency & Safety:
  • Guarded by a unique Symbol on Http2SubchannelCall.prototype so it is installed at most once.
  • Wrapped in a try / catch to fail open safely if @grpc/grpc-js module layout differs or changes in future releases.
  • Can be disabled at runtime by setting the environment variable SPANNER_DISABLE_HTTP2_STREAM_END_WORKAROUND=true.

Removal Path

This workaround is temporary and will be removed once upstream PR grpc/grpc-node#3082 is merged, released in @grpc/grpc-js, and adopted as a minimum dependency.

Benchmark Results

To evaluate the effect of avoiding redundant HTTP/2 stream closures in @grpc/grpc-js, we ran 30-minute side-by-side benchmarks on Google Compute Engine VMs against Cloud Spanner comparing main against this branch.

Test Environment & Workloads

• Infrastructure: GCE n2-standard VMs located in europe-north1-a connecting to Cloud Spanner instance spring-data-jpa via sidecar workload generator.
• Duration: 30 minutes per benchmark run.
• Steady Workload: 100 TPS baseline point-select queries running on n2-standard-2 (2 vCPUs).
• Bursty Workload: 2,000 TPS baseline point-select queries with periodic 3.8x bursts (7,600 TPS), running on n2-standard-4 (4 vCPUs) across 3 Node.js worker threads.
• Metrics Source: Google Cloud Monitoring (workload.googleapis.com/spanner_client_benchmarks/latency explicit histogram distribution).

---

1. Steady Point-Select Benchmark (100 TPS, 30m)

Metric	main	This PR	Delta
Total Operations	180,023	179,343	-680 (-0.3%)
Errors	0	0	0
Mean Latency	4.21 ms	3.57 ms	-0.64 ms (-15.3%)
P50 Latency (Interpolated)	3.92 ms	3.39 ms	-0.54 ms (-13.7%)
P90 Latency (Interpolated)	4.95 ms	4.23 ms	-0.71 ms (-14.4%)
P95 Latency (Interpolated)	5.80 ms	4.67 ms	-1.13 ms (-19.6%)
P99 Latency (Interpolated)	8.40 ms	6.65 ms	-1.75 ms (-20.8%)

Latency Distribution

Bucket	main Count (%)	This PR Count (%)
< 4.0 ms	100,391 (55.8%)	151,107 (84.3%)
4.0 – 6.0 ms	72,567 (40.3%)	25,483 (14.2%)
6.0 – 10.0 ms	6,461 (3.6%)	2,553 (1.4%)
> 10.0 ms	604 (0.3%)	200 (0.1%)

---

2. Bursty Channel-Scaling Benchmark (2,000 TPS baseline, 7,600 TPS bursts, 30m)

Metric	main	This PR	Delta
Total Operations	3,399,140	3,540,435	+141,295 (+4.2%)
Errors	0	0	0
Mean Latency	21.48 ms	11.06 ms	-10.42 ms (-48.5%)
P50 Latency (Interpolated)	4.98 ms	4.16 ms	-0.82 ms (-16.4%)
P90 Latency (Interpolated)	81.45 ms	19.47 ms	-61.98 ms (-76.1%)
P95 Latency (Interpolated)	117.56 ms	58.87 ms	-58.69 ms (-49.9%)
P99 Latency (Interpolated)	167.49 ms	126.73 ms	-40.77 ms (-24.3%)

Latency Distribution

Bucket	main Count (%)	This PR Count (%)
< 4.0 ms	1,170,432 (34.4%)	1,628,273 (46.0%)
4.0 – 6.0 ms	787,323 (23.2%)	879,082 (24.8%)
6.0 – 10.0 ms	434,426 (12.8%)	430,466 (12.2%)
10.0 – 50.0 ms	509,276 (15.0%)	400,843 (11.3%)
50.0 – 100.0 ms	261,614 (7.7%)	133,466 (3.8%)
> 100.0 ms	236,069 (7.0%)	68,305 (1.9%)
Total $\ge$ 50.0 ms	497,683 (14.6%)	201,771 (5.7%)

---

Summary of Findings

• Under steady baseline load (100 TPS), the workaround reduces mean latency by 0.64 ms, with 84.3% of queries finishing under 4 ms (vs 55.8% on main).
• Under bursty load, preventing redundant stream destructions eliminates synchronous V8 stack-trace generation for ERR_STREAM_ALREADY_FINISHED errors. This keeps the Node.js event loop responsive during bursts, reducing P90 latency from 81.45 ms to 19.47 ms and reducing requests queued $\ge 50$ ms from 14.6% to 5.7%.

[gemini-code-assist]

Code Review

This pull request introduces a defensive workaround to intercept redundant .end() calls on HTTP/2 streams, preventing unnecessary NodeError and stack trace allocations that degrade performance. The workaround is applied in the Spanner constructor and is accompanied by comprehensive unit tests. The review feedback highlights two important improvements: replacing a while loop with a for loop when uncorking to avoid potential infinite loops if uncork is mocked, and adding a defensive check for this to prevent a TypeError if the patched end method is called unbound.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a defensive workaround to address a performance issue in @grpc/grpc-js where redundant .end() calls on already-ended or destroyed HTTP/2 streams cause unnecessary NodeError allocations and V8 stack trace captures. The workaround patches stream.Duplex.prototype.end to short-circuit these redundant calls when no chunk or callback is provided, and is integrated into the Spanner client constructor. Comprehensive unit tests have been added to verify the workaround's behavior, including environment variable toggles, idempotency, and safety checks. I have no feedback to provide as there are no review comments.