Skip to content

xds: fail closed when ext_authz response processing throws - #13051

Merged
sauravzg merged 1 commit into
masterfrom
dev/sauravzg/extauthz-fail-open
Sep 11, 2026
Merged

xds: fail closed when ext_authz response processing throws#13051
sauravzg merged 1 commit into
masterfrom
dev/sauravzg/extauthz-fail-open

Conversation

@sauravzg

Copy link
Copy Markdown
Contributor

When the ext_authz filter processes a CheckResponse, CheckResponseHandler.handleResponse() only catches the checked HeaderMutationDisallowedException. An unchecked exception raised anywhere else in that path escaped AuthzCallbackObserver.onNext().

gRPC reacts to an exception from an application callback by cancelling the stream, which invokes onError(). There, failure_mode_allow sent the request on to the backend. The net effect is that an explicit PERMISSION_DENIED from the authorization server could be turned into an ALLOW.

The most accessible trigger is a header value that gRPC metadata cannot represent. HeaderValue.create() rejects anything outside horizontal tab, space and printable ASCII, while the value arrives in a proto string field carrying arbitrary UTF-8. An authorization server that echoes back a non-ASCII character in a header value is enough; a malicious server is not required.

failure_mode_allow is defined in terms of the authorization service being unreachable or returning an error. It does not cover a failure to process a response that the service successfully returned. Routing local processing errors through that policy is the underlying defect, so onNext() now handles its own failures and fails the call with INTERNAL rather than letting them reach onError().

Fixing this at the observer rather than at each individual throw site means the invariant holds for future call paths too. The orElseThrow() on the DENY branch is a second instance that was already reachable in principle and is now covered.

This change also makes a mutation targeting a gRPC-owned header key fail the RPC instead of being silently dropped.

Testing:

  • deny_withMalformedHeader_failOpen_doesNotReachBackend is the regression test. Without the fix it fails with the backend having received the request despite an explicit denial.
  • deny_withMissingStatus_failsCallWithInternal covers the orElseThrow() path; it previously asserted the exception escaped.
  • handleResponse_ok_grpcOwnedHeader_deniesCall covers the gRPC-owned header behavior, split out of the former combined binary-header test.

CheckResponseHandler.handleResponse() only catches the checked
HeaderMutationDisallowedException. An unchecked exception raised while
processing a CheckResponse -- for example HeaderValue.create() rejecting a
header value that is not valid ASCII -- escaped AuthzCallbackObserver.onNext().
gRPC then cancelled the stream and invoked onError(), where failure_mode_allow
sent the request to the backend, turning an explicit PERMISSION_DENIED into an
ALLOW.

failure_mode_allow covers the authorization service being unreachable or
returning an error. It does not cover a failure to process a response that the
service successfully returned, so onNext() now handles its own failures and
fails the call with INTERNAL instead of letting them reach onError().

Also fail the RPC, rather than silently dropping the mutation, when the authz
server attempts to mutate a gRPC-owned header.
@sauravzg
sauravzg merged commit 52f3f2b into master Sep 11, 2026
36 of 39 checks passed
@sauravzg
sauravzg deleted the dev/sauravzg/extauthz-fail-open branch September 11, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants