Skip to content

fix: Reuse IdP-issued client tokens until near expiry - #6687

Open
larrysingleton007 wants to merge 4 commits into
feast-dev:masterfrom
larrysingleton007:fix/oidc-client-token-reuse
Open

fix: Reuse IdP-issued client tokens until near expiry#6687
larrysingleton007 wants to merge 4 commits into
feast-dev:masterfrom
larrysingleton007:fix/oidc-client-token-reuse

Conversation

@larrysingleton007

Copy link
Copy Markdown
Contributor

What this PR does / why we need it

This is the client-side sibling of #6683. On the client_secret (client credentials / ROPC) branch, every outbound RPC builds a fresh auth manager and throws away the token it fetched:

  • get_auth_token constructs a new factory, manager, and OIDCDiscoveryService per call (client_auth_token.py).
  • All three interceptors call it per request: the gRPC interceptor on every intercepted call, the Arrow Flight middleware in sending_headers, and the HTTP wrapper on every session reuse.

So each request pays a discovery GET plus a token POST against the IdP for a token that is typically valid for an hour. Measured with counting mocks on master: 5 outbound calls produce 5 discovery GETs and 5 token POSTs. Beyond the added latency, a batch loop multiplies IdP load by request count, which can trip provider rate limits.

The fix caches issued tokens keyed by the token-request identity (discovery URL, client id and secret, username, password) and reuses each until shortly before expiry, taken from the token's own exp claim with the token response's expires_in as a fallback. After the change the same measurement performs 1 discovery GET and 1 token POST.

The cache is module level because the interceptors construct a fresh manager per RPC, so instance state cannot survive between calls. Only the client_secret branch is affected; static tokens, token_env_var, and mounted service account tokens were already cheap and are untouched.

A few details worth reviewer attention:

  • token_refresh_margin_seconds (default 30) is exposed on OidcClientAuthConfig rather than hardcoded, following the same request you made on fix: Reuse the OIDC JWKS client across requests #6683. It rejects non-positive values, since a zero or negative margin would allow reuse right up to or past expiry. It sits on the client config rather than the shared OidcAuthConfig because only clients fetch tokens from the IdP.
  • A token whose expiry cannot be determined (opaque, and no expires_in) is deliberately not cached, preserving today's per-call behavior rather than guessing a lifetime.
  • Errors from the IdP propagate before any cache write, so a failed fetch never poisons the cache.
  • Concurrent misses for the same identity may fetch in parallel and the last write wins. That costs at most a redundant fetch and never returns a wrong-identity token, which is still strictly better than the guaranteed fetch per request it replaces.

One interaction worth flagging: HttpSessionManager.get_session re-calls get_auth_token on every session cache hit, with a comment from #5895 explaining it does so "in case it expired". After this change that call returns a cached token rather than a freshly fetched one. The intent still holds, because the cache never returns a token with less than the margin remaining, but the mechanism changes and it seemed better to say so than to let a reviewer find it.

Testing: 7 new unit tests covering reuse until expiry, refusal to cache inside the refresh margin, the expires_in fallback for opaque tokens, no caching when expiry is unknowable, cache keying across distinct configs, the configurable margin changing reuse behavior, and rejection of non-positive margins. An autouse fixture resets the module cache around every test so no test inherits or leaks a cached token. 315 permissions tests pass; ruff and mypy are clean.

Which issue(s) this PR fixes

Fixes #6684

The client auth interceptors build a fresh manager for every outbound
RPC, so the client_secret flow paid a discovery GET plus a token POST
against the IdP per request and discarded the token. A batch loop
multiplies IdP load by request count and can trip IdP rate limits.

Cache issued tokens in a module-level store keyed by the token-request
identity, reusing each until 30 seconds before its expiry, read from the
token's exp claim with the token response's expires_in as fallback. A
token whose expiry cannot be determined is not cached, preserving the
per-call behavior for opaque tokens. Static, env-var, and mounted
service-account token sources are unaffected.

Measured with counting mocks: 5 outbound calls previously performed
5 discovery GETs and 5 token POSTs; now 1 and 1.

Signed-off-by: Larry Singleton <166439969+larrysingleton007@users.noreply.github.com>
Review feedback pattern from feast-dev#6683: the maintainer asked there for the
hardcoded JWKS cache tunables to move onto OidcAuthConfig, and this
change had the same shape of hardcoded constant.

Expose token_refresh_margin_seconds on OidcClientAuthConfig (rather than
the shared OidcAuthConfig, since only clients fetch tokens from the IdP),
defaulting to 30 and rejecting non-positive values: a zero or negative
margin would let a token be reused right up to or past its expiry.

Also from review: soften a comment that claimed a reused token 'cannot'
expire in flight, which is a fixed heuristic rather than a guarantee
under clock skew; state in the docstring that near-expiry tokens are not
cached either; and rewrite a tuple-with-conditional return that two
reviewers had to verify empirically before accepting it was not a
precedence bug.

Signed-off-by: Larry Singleton <166439969+larrysingleton007@users.noreply.github.com>
@larrysingleton007
larrysingleton007 requested a review from a team as a code owner July 31, 2026 17:15
@codecov-commenter

codecov-commenter commented Jul 31, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 46.81%. Comparing base (a1e6fc2) to head (5aff938).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...sions/client/oidc_authentication_client_manager.py 97.50% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6687      +/-   ##
==========================================
+ Coverage   46.77%   46.81%   +0.04%     
==========================================
  Files         414      414              
  Lines       50191    50229      +38     
  Branches     7181     7188       +7     
==========================================
+ Hits        23475    23513      +38     
+ Misses      25077    25074       -3     
- Partials     1639     1642       +3     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.14% <97.56%> (+0.04%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/permissions/auth_model.py 100.00% <100.00%> (ø)
...sions/client/oidc_authentication_client_manager.py 82.29% <97.50%> (+9.41%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4efb86c...5aff938. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Client-side OIDC auth pays a discovery fetch and a token fetch per outbound request

2 participants