feat(openfeature): emit server-side EVP flagevaluation - #11639
Conversation
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d4244f8ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
83ac4c4 to
81ed6f1
Compare
…t logging At shutdown there is nowhere to surface the error usefully, and each writer already logs internally on failure. Debug-logging from the catch blocks was misleading noise. True swallow (Exception ignored) matches the intent of the original suggestion. Co-Authored-By: Claude <noreply@anthropic.com>
…ingSystem.stop() Replaces four identical try/catch blocks with a single closeQuietly helper. SpanEnrichmentWriter gains AutoCloseable (it already had a close() method) so all four resources share the same call site. Co-Authored-By: Claude <noreply@anthropic.com>
…tionAggregator EVAL_SCALE_ prefix was ambiguous — it read as a runtime scale factor rather than design-time sizing assumptions. Split into two named groups: - EXPECTED_* for the design assumptions (flag count, users per flag, etc.) - *_SIZING_BASIS for the derived intermediate values - Inline comments on GLOBAL_CAP and DEGRADED_CAP explain they are the nearest powers of two above the respective sizing bases. Co-Authored-By: Claude <noreply@anthropic.com>
The multi-pass loop with Thread.yield() between passes was a heuristic for catching producers mid-enqueue during shutdown. A single poll loop already drains everything in the queue at that point. For events that race past the drain, close() sweeps the queue after joining the worker and counts any remainder as an observable drop — making extra passes redundant. Removed SHUTDOWN_DRAIN_PASSES constant and the loop. Co-Authored-By: Claude <noreply@anthropic.com>
The lazy Supplier overload and contextAttributes() accessor were removed in 29b974d, breaking compileTestJava. Drop the two tests that covered the deleted lazy path and remove the contextAttributes() assertions from the remaining tests. Co-Authored-By: Claude <noreply@anthropic.com>
…verload The lazy Supplier constructor was removed in 29b974d; pass attrs directly. Also rename nextLazyEvent -> nextEvent in FlagEvaluationHotPathBenchmark. Co-Authored-By: Claude <noreply@anthropic.com>
Commit 29b974d removed the lazy-supplier path and took two tests with it, dropping FlagEvaluationWriterImpl branch/instruction coverage below the 90% Jacoco threshold on Java 8. Add two replacement tests: - scoConstructorCreatesUsableWriter: exercises the public SCO constructor - countContextTruncatedAccumulatesPerReason: exercises countContextTruncated Co-Authored-By: Claude <noreply@anthropic.com>
The pre-queue guard methods were only exercised through mocks in the hook test, leaving FlagEvaluationWriterImpl branch coverage at 0.844 and failing jacocoTestCoverageVerification (threshold 0.9). Add a direct test that observes both branches of hasCapacityForEnqueue and the countPreQueueOverflow counter surfacing as a queue_overflow drop metric. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sabrenner
left a comment
There was a problem hiding this comment.
llmobs span mapper changes lgtm - i believe this follows our node.js & python limits as well
…ions events (#12042) * Parse observeFullEvaluationData and hash targeting_key in flagevaluations events Adds the top-level observeFullEvaluationData boolean to the UFC model, plumbs it through to the EVP flagevaluation event serializer, and gates PII handling on it: when the flag is absent/false the targeting key is SHA-256 hashed (sha256_<hex>) and the raw evaluation context is omitted from the wire; when true the raw targeting key and context are emitted. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Extract hashed targeting key prefix into a named constant Replace the inline "sha256_" literal with a documented HASHED_TARGETING_KEY_PREFIX constant describing the cross-SDK wire contract for privacy-preserving hashed targeting keys. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Test observeFullEvaluationData parsing edge cases Parameterize the true/false config-parsing assertions with @valuesource and add a test locking in the fail-closed behaviour for an explicit JSON null: malformed config is rejected so full evaluation data is never observed off the back of it. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData per bucket at aggregation time The flush-time read of FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() was a TOCTOU bug: CURRENT_CONFIG could be overwritten by a later RC update between when an evaluation happened and when the batch flushed, so events could be emitted under the wrong environment's consent (the system test observed a targeting key hashed even though the active UFC said observeFullEvaluationData=true). Capture consent when the evaluation is folded into its EvalBucket instead. On merge the value is folded with AND, so any no-consent evaluation in a bucket's lifetime sinks the whole bucket to hashed/omitted (fail-closed). buildEventList now reads bucket.observeFullEvaluationData rather than the gateway. The gateway accessor is retained; it is read at aggregation time. Adds a writer-level regression guard (a bucket aggregated under consent-off stays hashed even if the gateway later reports consent-on) plus aggregator fold tests, and an end-to-end parse->dispatch->flush test. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Capture observeFullEvaluationData consent at evaluation time Snapshot the PII consent flag on the evaluation thread (in the OpenFeature hook) and carry it on FlagEvalEvent, instead of reading the gateway when the event is aggregated/flushed. This pins the hashed-vs-raw decision to the configuration active at evaluation time, closing a one-directional leak window where a later Remote Config update could retroactively apply a different environment's consent to already-collected evaluations. Aggregation and flush now read event.observeFullEvaluationData and never consult the gateway; the AND-fold across a bucket's evaluations is unchanged (any no-consent evaluation sinks the bucket to hashed/omitted). Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Bind observeFullEvaluationData consent to the evaluator's configuration Address PR #12042 review feedback (Codex P1, leoromanovsky, dd-oleksii): the FlagEvalLoggingHook was reading observeFullEvaluationData from FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() at hook-fire time, which races against a Remote Config swap of CURRENT_CONFIG that happens after DDEvaluator.evaluate() captured its own ServerConfiguration reference. That race can retroactively mark an evaluation performed without consent as consented and leak the raw targeting key / context. DDEvaluator now stamps the boolean directly from the ServerConfiguration it used, onto every ProviderEvaluation via ImmutableMetadata under key "dd.observe_full_evaluation_data". The hook reads consent from that metadata and no longer queries the gateway. Missing metadata (PROVIDER_NOT_READY or a non-DD provider) → false, the privacy-preserving default. The gateway's isObserveFullEvaluationDataEnabled() accessor is removed since its only real caller was the hook and re-adding it would re-open the race. Adds regression tests: hook honours consent metadata (true/false/absent) and ignores a gateway value that disagrees; evaluator stamps the correct boolean on the FLAG_NOT_FOUND path and omits metadata when it holds no config. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Pass the boolean, not the ServerConfiguration, into error/resolveVariant Follow-up to the previous commit: the private error() and resolveVariant() helpers only ever read one field off the ServerConfiguration (observeFullEvaluationData), so pass the boolean directly instead of the whole config. Keeps the internal API narrow and removes the incidental coupling these helpers had to the UFC. While here, PROVIDER_NOT_READY now stamps consent as the privacy-preserving false rather than omitting the metadata. Same on-the-wire outcome the hook would have produced, but the invariant "every DD-produced evaluation carries dd.observe_full_evaluation_data" is now unconditional, which is easier to reason about. The two error() overloads collapse to one (the (String) null casts at call sites disappear along with them). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop the dd. prefix on the evaluation-metadata consent key The key is only ever read by FlagEvalLoggingHook one line later — it never lands on the wire, so it doesn't need the "dd." namespacing that "dd.eval.timestamp_ms" has (that key is re-emitted onto spans). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Trim verbose comments around the observeFullEvaluationData plumbing The race-vs-CURRENT_CONFIG backstory is captured in the previous commits' messages; the code only needs the forward-looking invariants (metadata is source of truth, missing key = false, DD-produced evaluations always stamp). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation context when aggregating consent-off evaluations Address PR #12042 review from leoromanovsky (escalated Codex P2 → P1): on the protected path (observeFullEvaluationData=false) the serializer drops the evaluation context, but the aggregator was still running it through pruneContext + canonicalContextKey and keying every full-tier bucket on it. A high-cardinality field on the evaluation context (request_id, timestamp, correlation id) would fragment buckets that emit byte-identical wire rows, blow out PER_FLAG_CAP (10k) inside one flush window, and force subsequent evaluations into the degraded tier — which drops the targeting key entirely. On the protected path aggregate() now uses ctxKey="" and stores prunedAttrs=null, so different contexts for the same subject collapse into one bucket. The targeting key stays in the aggregation identity, so different subjects still hash to different buckets. The consent-on path is unchanged. Regression tests: protected path collapses differing contexts for one subject; protected path still separates distinct subjects; full path still splits on context. Existing tests that exercise pruneContext / context-differentiation were updated to use consent=on (that's the code path they actually cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Skip evaluation-context capture on the hook hot path when consent is off Companion to the aggregator fix: with observeFullEvaluationData=false the evaluation context is dropped on emit and no longer influences aggregation, so there is no reason to snapshot it on the evaluation thread. The hook now branches on consent up front — the protected path enqueues an event with an empty materialized attrs map (no map copy of the OpenFeature context, no Supplier<Map> allocation, no lambda instance), while the consent-on path is unchanged. Grep confirms the only production consumer of FlagEvalEvent.contextAttributes / FlagEvalEvent.attrs is FlagEvaluationAggregator.aggregate, which already skips them on the protected path. Regression test: mutating the EvaluationContext after finallyAfter returns still yields empty attrs on the enqueued event — proves the hook never snapshotted it. Two existing tests that exercise the snapshot mechanism were switched to pass consent-on metadata (that's the code path they cover). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Include observeFullEvaluationData in the aggregation bucket key Bucket keys should cover every dimension the emitter will branch on. The serializer branches on observeFullEvaluationData (hashes the targeting key and drops the context when off), so two evaluations that differ only in consent produce different wire rows and must not share a bucket. Before this change they could: same subject, same flag, same empty context would land under the same FullKey regardless of consent, and the AND-fold would silently downgrade a consent-on evaluation to the protected wire shape because a nearby consent-off event merged into its bucket first. No PII leak (fail-closed direction), but arrival-order-dependent semantics and a lost raw-context row. Add observeFullEvaluationData to FullKey / DegradedKey (equals + hashCode). The AND-fold on bucket.observeFullEvaluationData stays as defensive belt- and-suspenders; every event merging into a bucket now carries the matching consent value by construction. Regression test: two events identical except for consent land in two full- tier buckets, one consent-on and one consent-off. Updated the previous "fold to false on mixed consent" test to reflect the new invariant. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add consent metadata to ProviderTest flag-eval-logging hook route test The test asserts that context attributes flow through the logging hook, but the mock metadata omitted the observe-full-evaluation-data flag, so the hook took the privacy-preserving path and dropped context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Redact error messages when observeFullEvaluationData is off Exception messages from the evaluator's outer catch blocks (NumberFormatException, generic Exception) can echo raw evaluation-context values verbatim — for example a GT rule on "id" with a PII-shaped targeting key produced error.message="For input string: \"jane.doe@...\"" on the wire regardless of consent, defeating the PR's own PII guard. Drop the message at DDEvaluator.error() when consent is off, and add a hook-layer fallback that substitutes ErrorCode.name() so operators keep a stable signal (e.g. "TYPE_MISMATCH") even when a third-party provider hands us a raw message. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Exercise every consent-stamp code path in DDEvaluatorTest The only observeFullEvaluationData assertions were on error paths (FLAG_NOT_FOUND, PROVIDER_NOT_READY), leaving the success-path stamp in resolveVariant and the DISABLED/DEFAULT stamps in consentMetadata uncovered — line 448 could be deleted or hardcoded to either value and every existing test would still pass. Add symmetric consent-on/consent-off tests for each of resolveVariant, DISABLED, and DEFAULT so any mutation (delete / hardcode true / hardcode false) flips at least one assertion. Rename the previously misleading …OnSuccess test to reflect what it actually exercises (FLAG_NOT_FOUND error via error()). Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop consent from DegradedKey to reclaim effective DEGRADED_CAP Two degraded buckets differing only in observeFullEvaluationData emit byte-identical wire JSON — the degraded serializer (fromBucket with isFullTier=false) drops the targeting key and context regardless of consent — so the consent dimension in DegradedKey halved effective DEGRADED_CAP for zero wire fidelity gain. FullKey correctly keeps consent (the full-tier serializer branches on it for raw-vs-hashed targeting key and context inclusion). Mixed-consent events now merge into one degraded bucket. The AND-fold on EvalBucket.observeFullEvaluationData still runs and collapses to false whenever any consent-off event lands in a mixed bucket; benign because the value has no downstream effect for degraded rows. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Tolerate malformed observeFullEvaluationData in UFC parse Before this change ServerConfiguration.observeFullEvaluationData was a primitive boolean — Moshi's reflective adapter rejected the entire UFC whenever the JSON value was null or wrong-typed. Agentless swallows the IOException at DEBUG, so a pod starting after a malformed message had no last-known-good, stranded every flag on PROVIDER_NOT_READY, and served defaults forever. Fail-closed on privacy shouldn't cascade into fail-closed on availability. Box the field to Boolean so null tolerates naturally, register a LenientBooleanAdapter that maps wrong-typed values to null as well, and read via Boolean.TRUE.equals(...) at the DDEvaluator so null falls to the privacy-preserving default. The lenient adapter only intercepts Boolean (not primitive boolean), so mandatory fields like Flag.enabled keep their strict parse; the only other Boolean it touches is Allocation.doLog, which is already read as `!= null && doLog`. Reversed the earlier RejectsExplicitNull test — it had locked in the buggy behaviour — into a family of tolerance tests for null / stringified / numeric. Added a DDEvaluator test that a config with a null consent field evaluates without NPE and stamps the privacy-preserving default. Generated with Claude Code Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Set observeFullEvaluationData=true for the NaN-poison flush test The consent-off short-circuit in FlagEvaluationEvent.fromBucket drops the raw context before Moshi encodes it, so a NaN in the attrs never reaches the encoder and the flush succeeds. That defeated the intent of encodeFailureClearsAggregatorSoLaterFlushesRecover, which must observe a real encode failure to prove the aggregator is cleared. Co-Authored-By: Claude <noreply@anthropic.com> * Cover LenientBooleanAdapter read-only and qualifier paths The per-class JaCoCo gate (0.9 minimum, gradle/jacoco.gradle) failed on the new adapter: toJson was never invoked (20/25 instructions) and the factory's !annotations.isEmpty() short-circuit never evaluated true (3/4 branches). Neither path is reachable through the parse-driven tests in JsonApiUfcResponseParserTest. Mirror the tests the sibling FlagMapAdapter and DateAdapter already have. The primitive-boolean assertion documents the guard that keeps this leniency off mandatory fields like Flag.enabled. Environment: Datadog workspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
📢 FYI I merged #12042 into this branch so we have PII protected when we merge to
|
…ng/canonicalization Address review feedback from @AlexeyKuznetsov-DD on the FlagEvaluationAggregator: - Replace String.format("%08x", ...) in appendLengthDelimited with a StringBuilder-based zero-padded hex writer. Format on the per-context-field path allocates a Formatter and boxes the length argument on every call. - Rewrite FullKey.hashCode and DegradedKey.hashCode with HashingUtils.hash / addToHash. Objects.hash allocates an Object[] per call and boxes the boolean dimensions; both keys are looked up on every event dispatch. Co-Authored-By: Claude <noreply@anthropic.com>
| try { | ||
| initializeSystem(sco, config); | ||
| } catch (final RuntimeException | Error e) { | ||
| STARTED = false; | ||
| throw e; | ||
| } |
There was a problem hiding this comment.
This code looks like a duplicated (2 same places).
And also probably is missing of safe closing of exposureWriter and configService.
It is like partially initialized state did not rolled back.
There was a problem hiding this comment.
Good catch on both points, thank you. Fixed in 2c0d326.
Duplication: the identical try { initializeSystem } catch { STARTED = false; throw } block in start() and activateAgentless() is now a single private helper, initializeOrRollBack.
Partially initialized state: you are right that the rollback was incomplete. The old handler only cleared STARTED. If initializeSystem threw after initialize() had already published CONFIG_SERVICE and EXPOSURE_WRITER (for example from evalWriter.start() or SpanEnrichmentWriter.init()), those two stayed open and unreachable, and the gateway kept its enqueue flag set. The helper now calls stop() on failure, which closes every published resource, removes any activation listener, clears the gateway flags, and clears STARTED. stop() is synchronized on the same class monitor the caller already holds, so the reentrant call is safe, and it was already idempotent, so it tolerates being called when only part of the state exists.
While there I fixed one more instance of the same problem: FLAG_EVAL_WRITER was assigned after evalWriter.start(), so a writer whose start() failed was unreachable and no rollback could close it. The assignment now comes first.
The pre-existing inner rollback in initialize() for the config-service and exposure-writer pair is unchanged and still correct. The new outer rollback covers the wider window that opens after initialize() returns.
Added failedStartRollsBackPartiallyInitializedState to cover it: a failed start must leave no pending activation listener, no gateway writer, enqueue disabled, and a restartable subsystem.
The failure handler only cleared STARTED. If initializeSystem threw after initialize() had published CONFIG_SERVICE and EXPOSURE_WRITER, those stayed open and unreachable, and the gateway kept its enqueue flag. Route both start paths through one initializeOrRollBack helper that calls stop() on failure, which closes every published resource, removes any activation listener, and clears the gateway flags. This also removes the duplicated try/catch shared by start() and activateAgentless(). Also publish FLAG_EVAL_WRITER before evalWriter.start(), so a writer whose start fails is still reachable by the rollback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FlagEvaluationHotPathBenchmark built its events with the convenience constructor, which defaults observeFullEvaluationData to false. Under consent-off the aggregator drops attrs and skips canonicalContextKey, so every field-count profile measured the same scalar-only work: 10 fields and 250 fields both landed near 30 ns. Pass consent true so the benchmark exercises canonicalization again. The profiles now spread from 498 ns at 10 fields to 17.3 us at 250 fields. Add FlagEvalHookHotPathBenchmark in feature-flagging-api to cover the evaluation-thread cost the lib benchmark cannot reach. It splits total inline cost, the bounded context copy alone, and the consent-off floor across flat, nested, and list context shapes. This required enabling the me.champeau.jmh plugin on the module, which had no benchmark source set. Also refresh the lib benchmark javadoc, which still described an attribute-supplier API that no longer exists and claimed the hook cost was unmeasured anywhere. Co-Authored-By: Claude <noreply@anthropic.com>
System-tests validation against this branch (local run)Ran the full FFE system-tests surface against this branch to confirm the EVP flagevaluation work passes end-to-end and doesn't regress existing Java FFE behavior. Tracer under test: How it was builtFFE needs three jars in ./gradlew :dd-java-agent:shadowJar \
:products:feature-flagging:feature-flagging-api:jar \
:dd-trace-api:jar
# copy ONLY the plain jars into system-tests/binaries/
# (install_ddtrace.sh hard-fails if its glob matches >1, and gradle
# also emits -sources / -javadoc / -jmh variants)
./build.sh -i weblog -l java -w spring-boot
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
./run.sh PARAMETRIC --library java tests/parametric/test_ffe/ResultsEnd-to-end (
All 11 EVP classes were confirmed individually from This covers the PII targeting-key hashing vectors, aggregation/counting, context bounds, runtime defaults, and the degradation/overflow path. Parametric ( Notes on the non-failures
Jar provenanceVerified the tests actually exercised this branch's code rather than a released artifact — SHA-256 identical across gradle output →
Additional confirmation: the weblog image contains 🤖 Generated with Claude Code |
Manual validation against staging (ffe-dogfooding, Java SDK)Ran this branch end-to-end through the SetupDD_TRACE_JAVA_PATH=~/dd/dd-trace-java scripts/prepare-local-java.sh
DD_TRACE_JAVA_PATH=~/dd/dd-trace-java docker compose \
-f docker-compose.yml -f local/docker-compose.java.yml \
up --build -d mock-intake otlp-intake datadog-agent app-java evaluatorLocally built ResultsProvider + evaluations — 170 evaluations, 100% success, 0 failures.
flagevaluation EVP events land on
Worth noting for anyone reproducing:
Consent ON: {
"targeting_key": "user-0gu401kuar6q",
"context": {"evaluation": {"country": "US", "plan": "pro", "version": "1.0.0"}},
"flag": {"key": "ffe-dogfooding-float-flag"},
"variant": {"key": "-3.14"},
"allocation": {"key": "allocation-override-b84f868b6ff5"},
"evaluation_count": 1
}Consent OFF: {
"targeting_key": "sha256_a320ad9bdc04de267751bb3742b3814de63aea1b41c3bbc62eca83c09f59ad46",
"flag": {"key": "ffe-dogfooding-string-flag"},
"variant": {"key": "control"},
"allocation": {"key": "3baabb3c-2471-4ef8-b04e-1ab20790abe9"},
"evaluation_count": 1
}Audited all 74 events captured under consent-OFF:
The functional payload survives redaction in both modes: allocation key, variant, and aggregation counters are intact under consent-OFF, so only subject-identifying data is dropped. The toggle propagated live in both directions without an app restart — flipping consent off moved the verdict to Staging was restored to its original state afterward and verified byte-for-byte against a pre-change |
close() interrupts the worker thread to break it out of queue.poll(100ms). The worker then falls into the finally block in run(), which calls drainAndFlush() to do the final flush that close() exists to guarantee. The interrupt flag is still set at that point. That flush does socket I/O, and OkHttp fails fast on a thread whose interrupt flag is set. The resulting IOException is swallowed by the broad catch in flush(), whose finally then calls aggregator.clear(). So every aggregated evaluation in the final flush window was discarded on every clean shutdown, and the loss was invisible: it surfaced as an error log rather than a drop metric. close()'s post-join sweep does not cover this. The sweep counts events still sitting in the queue. These rows had already been drained out of the queue into the aggregator before flush() threw, so the sweep cannot see them. Save and clear the interrupt flag around drainAndFlush(), then restore it so the thread still exits with the correct interrupt status. The bug needed both halves to appear, and neither is wrong alone: the interrupt came from 5c6cb2b ("Make flagevaluation enqueue lock-free"), which needs it to wake the worker now that the lock no longer orders offers against the drain; the finally-block drain came from 5f88cad. The existing closeDrainsAndFinalFlushesQueuedEvents test passes against the bug because a mocked publisher ignores the interrupt flag - only a real socket reacts to it. The new test therefore asserts on the flag state at the moment post() is invoked. Verified it fails without the production change: "expected: <false> but was: <true>". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
🟢 NOTE TO REVIEWERS
I've chosen to keep this PR on the larger side in terms of "lines of code" as a test. The commits are deliberately layered in a narrative style. Each one is equivalent to what we would normally do with a "stacked" PR, but this preserves the overall view of the feature. Please review one commit at a time or all together.
If this review mechanism is not satisfactory, please let me know!
Stack Position
You are here: the second Java layer, stacked directly on #11892. It adds aggregate
flagevaluationEVP independently of whether UFC arrived through agentless HTTP or Agent Remote Configuration.flowchart LR subgraph JAVA["dd-trace-java"] J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"] end subgraph SYSTEM["system-tests"] ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"] ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"] ST2 --> NEXT["Next drafts<br/>Enable Java side effects"] end subgraph DOGFOOD["ffe-dogfooding"] DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"] end J1 --> STM1 J1 --> DOG0 J2 --> NEXT J2 --> DOG1 STM1 --> GREEN["Java proof<br/>both sources × side effects"] NEXT --> GREEN DOG1 --> GREEN classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class J2 current;Motivation
Customers need consistent server-side feature-flag evaluation visibility across supported runtimes so rollout behavior can be correlated with application behavior in APM and Event Platform. This Java contribution adds that server-side
flagevaluationsignal for Java OpenFeature evaluations while preserving the existing OTelfeature_flag.evaluationspath and the existing exposure telemetry path.High Priority Changes and Decisions
These are the design points I would want reviewed most closely.
EVP routing uses the Agent-advertised proxy prefix, not a hard-coded v2/v3/v4 path. The SDK keeps the track route as
/api/v2/flagevaluation, but builds it under the proxy prefix discovered from the Agent. In current staging dogfooding that resolves to/evp_proxy/v4/api/v2/flagevaluation.Flagevaluation reuses the existing Event Platform publisher path. This keeps delivery aligned with existing Agent discovery, headers, lifecycle, and compression controls instead of adding a Java-specific HTTP writer. Response compression is disabled for this track to match the merged Go behavior.
The OpenFeature hook runs inline, and under consent-on the context copy — not scalar extraction — dominates its cost.
finallyAfterextracts scalar evaluation metadata, then copies the caller's evaluation context viaDDEvaluator.copyPrunedContext, then does a non-blocking enqueue. Only aggregation and posting are deferred to the worker thread. This is the main correctness/performance boundary.The copy is bounded rather than unconditional.
copyPrunedContextapplies every retained-size cap inline (field count, key length, value length, list width, structure width, depth), so work is proportional to what is kept, not to what the caller supplied. It also early-returns on an empty context, so a context with no attributes costs ~15 ns rather than allocating a cycle-detection set.Crucially, the copy only happens under consent-on. When
observeFullEvaluationDatais false — the privacy-preserving default — the context is dropped on emit and never consulted by the aggregator, so the hook skips the copy entirely. Measured onFlagEvalHookHotPathBenchmark, that protected path is a flat ~7 ns regardless of context shape, while consent-on ranges from ~29 ns (empty context) to ~3.8 µs (10 nested structures × 10 fields). The javadoc previously claimed the hook did "ONLY cheap scalar extraction," which understated the consent-on cost; it has been corrected in code and quantified below.The copy itself is unavoidable on the consent-on path:
EvaluationContextis caller-owned and mutable, so its values must be captured before the event is handed to the writer thread. Callers who enable full evaluation data and pass large or deeply nested contexts should expect low-single-digit-microsecond inline cost per evaluation.The worker emits the existing batched flagevaluation contract. One flush produces a
FlagEvaluationsRequestwith top-levelcontextand aflagEvaluationsarray; this does not use a separate/batchedflagevaluationsroute.Aggregation keys are limited to schema-visible fields. The aggregate dimensions are flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned context. OpenFeature
reasonis intentionally not a hidden aggregate key because it is not serialized to the worker contract.targeting_keyis the single identity field for the event. The hook removes duplicatetargetingKeyfromcontext.evaluationso the same identity is not encoded twice.Cardinality/backpressure behavior is intentionally lossy but counted. The writer uses full-fidelity buckets first, degraded buckets without targeting key/context after cap pressure, then counted drops if both tiers or payload limits are exhausted.
Event time and send time are separate. Each aggregate row preserves
first_evaluationandlast_evaluationbounds, while the payloadtimestampis the flush time.Other Changes
flagevaluationpath behindDD_FLAGGING_EVALUATION_COUNTS_ENABLEDwhile leaving the existing OTelfeature_flag.evaluationshook in place.FlagEvalHookHotPathBenchmark(feature-flagging-api) — the evaluation-thread cost offinallyAfteracross flat/nested/list context shapes, split into total inline cost (hookFinallyAfter), the bounded context copy alone (contextCopy), and the consent-off floor (hookFinallyAfterConsentOff). Adding this required enabling theme.champeau.jmhplugin onfeature-flagging-api, which had no benchmark source set.FlagEvaluationHotPathBenchmark(feature-flagging-lib) — writer queue mechanics and worker-thread aggregation.evalThreadCapturemethod is renamedwriterEnqueueto stop implying it covered that cost. The hook lives infeature-flagging-apiand its OpenFeature context types are not on the lib's classpath, which is why the inline cost has to be measured in a second module.FlagEvaluationHotPathBenchmarknow builds its events withobserveFullEvaluationData=true. Under the consent-off default the aggregator drops attrs and skipscanonicalContextKeyentirely, so every field-count profile collapsed to the same scalar-only cost and the benchmark silently stopped measuring canonicalization.Commit Guide
LOC is rename-aware
git diff-tree -M --numstatfor each commit against its parent.571938b6c74949ed7ee44aa06476b7ac126fb44a7a43658da64ec77cae036fa7ade3e4986a9af6bef1c37aba16115a407b76cc3b21c0c231d7cb5116FlagEvaluationsRequestpayloads, split oversized bodies, degrade oversized rows, and count drops.e43f93acd017d77854205f88cadeb2597e97d10b58df936165acc6e2194096df04f1beedb264e7d890ed6cb556c73cafdb465b1456b3fbValidation Evidence
Hot-Path Cost (JMH)
JDK 11.0.30, aarch64, 3×2s warmup / 5×1s measurement, single fork. Relative magnitudes are the point, not absolute numbers.
Evaluation-thread (inline) cost
./gradlew :products:feature-flagging:feature-flagging-api:jmh -PjmhIncludes=FlagEvalHookHotPathBenchmarkhookFinallyAfterconsent-on (ns/op)contextCopyalone (ns/op)hookFinallyAfterConsentOff(ns/op)flat/0attrsflat/10attrsflat/100attrsnested/10structs_10fieldslist/10lists_10itemsReadings:
copyPrunedContextearly-returns before allocating the cycle-detectionIdentityHashMap, so there is no fixed snapshot tax on contexts with nothing to copy. An earlier revision of this PR paid ~119 ns here; that has been fixed.flat/100attrs,nested/10structs_10fields, andlist/10lists_10itemsall carry 100 leaf values, but inline cost spans 1.5–3.8 µs. Nested structures are worst (~2.5× flat) because each allocates an inner map plus a structure wrapper on top of the leaf copies.¹
contextCopymeasuring marginally abovehookFinallyAfteris run-to-run variance on what is effectively the same work — the two are within ~5% of each other — not a negative-cost hook.Writer and worker cost
./gradlew :products:feature-flagging:feature-flagging-lib:jmh -PjmhIncludes=FlagEvaluationHotPathBenchmarkwriterEnqueue(ns/op)workerAggregate(ns/op)typical/100flags_50users_10fieldsstress/10flags_1000users_250fieldsscale/2500flags_500users_20fieldsReadings:
Local Test Gates
origin/master(ac29db2316)::communication:test:products:feature-flagging:feature-flagging-api:test:products:feature-flagging:feature-flagging-agent:test:products:feature-flagging:feature-flagging-lib:test:products:feature-flagging:feature-flagging-lib:jmhClassesBackendApiFactoryTest,DDAgentFeaturesDiscoveryTestDDEvaluatorTest,ProviderTest,FlagEvalLoggingHookTestFeatureFlaggingSystemTestFeatureFlagEvpPublisherTest,FlagEvaluationAggregatorTest,FlagEvaluationPayloadsTest,FlagEvaluationWriterImplTest./gradlew :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-lib:jacocoTestReport :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification./gradlew spotlessApply./gradlew spotlessCheck./gradlew :products:feature-flagging:feature-flagging-lib:spotlessCheck./gradlew :communication:forbiddenApis :dd-trace-core:forbiddenApis :internal-api:forbiddenApis :telemetry:forbiddenApis :products:feature-flagging:feature-flagging-api:forbiddenApis :products:feature-flagging:feature-flagging-agent:forbiddenApis :products:feature-flagging:feature-flagging-lib:forbiddenApisgit diff --checkpassed.Dogfooding App
ffe-dogfoodingJava artifacts from this localdd-trace-javastack withscripts/prepare-local-java.sh.dd-openfeatureanddd-java-agentartifacts plus the real backend EVP path.PROVIDER_READY.ffe-dogfooding-string-flagthrough the Java dogfooding app 15 times total: 5 evaluations for each targeting key:java-restack4-20260702T042247Z-alphajava-restack4-20260702T042247Z-bravojava-restack4-20260702T042247Z-charlievariant_1, allocationallocation-override-392dd7c149f8, servicejava, and evaluation reasonSTATIC.http://datadog-agent:8126/evp_proxy/v4/api/v2/flagevaluation, both returning202.Staging End-To-End
eventplatform.system.track(TRACK => 'flagevaluation')returned 3 aggregated rows for the exact targeting keys above.flag.key=ffe-dogfooding-string-flagvariant.key=variant_1allocation.key=allocation-override-392dd7c149f8evaluation_count=5System Tests
6b7aa4273d:TEST_LIBRARY=java ./run.sh +v FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_evp.py8 passed in 80.08s(Library: java@1.64.0-SNAPSHOT+6b7aa4273d,Weblog variant: spring-boot).Integration Addendum
JAVA-01 is the direct PR base and is preserved as an ancestor through signed merge commit
d5b90bad24.FeatureFlaggingSystemnow selects the configured source first, then starts exposure and aggregate-evaluation writers independently for agentless, Remote Configuration, and reserved offline source modes.Validation Evidence
dd-java-agent,dd-trace-api, anddd-openfeature: pass.CI Packaging Decision
Any custom Java system-test build must install all three artifacts. Supplying only
dd-java-agentanddd-trace-apileaves the weblog on the publisheddd-openfeatureimplementation and does not exercise the PR head.