Skip to content

feat(openfeature): emit server-side EVP flagevaluation - #11639

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 117 commits into
masterfrom
leo.romanovsky/ffl-2446-evp-flagevaluation-java
Aug 12, 2026
Merged

feat(openfeature): emit server-side EVP flagevaluation#11639
gh-worker-dd-mergequeue-cf854d[bot] merged 117 commits into
masterfrom
leo.romanovsky/ffl-2446-evp-flagevaluation-java

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

🟢 NOTE TO REVIEWERS

I've chosen to keep this PR on the larger side in terms of "lines of code" as a test. The commits are deliberately layered in a narrative style. Each one is equivalent to what we would normally do with a "stacked" PR, but this preserves the overall view of the feature. Please review one commit at a time or all together.

If this review mechanism is not satisfactory, please let me know!

Screenshot 2026-07-01 at 8 14 47 PM

Stack Position

You are here: the second Java layer, stacked directly on #11892. It adds aggregate flagevaluation EVP independently of whether UFC arrived through agentless HTTP or Agent Remote Configuration.

flowchart LR
    subgraph JAVA["dd-trace-java"]
        J1["JAVA-01 · #11892<br/>Agentless + RC sources"] --> J2["JAVA-02 · #11639<br/>Aggregate evaluation EVP"]
    end

    subgraph SYSTEM["system-tests"]
        ST1["ST-01 · #7298<br/>Mock agentless backend"] --> ST2["ST-02 · #7299<br/>Side-effect contracts"]
        ST2 --> STM1["ST-M01 · #7300<br/>Enable Java configuration"]
        ST2 --> NEXT["Next drafts<br/>Enable Java side effects"]
    end

    subgraph DOGFOOD["ffe-dogfooding"]
        DOG0["DOG-00 · #92<br/>Agentless evaluation baseline"] --> DOG1["DOG-01 · #93<br/>Side-effect conduit"]
    end

    J1 --> STM1
    J1 --> DOG0
    J2 --> NEXT
    J2 --> DOG1
    STM1 --> GREEN["Java proof<br/>both sources × side effects"]
    NEXT --> GREEN
    DOG1 --> GREEN

    classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111;
    class J2 current;
Loading

Motivation

Customers need consistent server-side feature-flag evaluation visibility across supported runtimes so rollout behavior can be correlated with application behavior in APM and Event Platform. This Java contribution adds that server-side flagevaluation signal for Java OpenFeature evaluations while preserving the existing OTel feature_flag.evaluations path and the existing exposure telemetry path.

High Priority Changes and Decisions

These are the design points I would want reviewed most closely.

  • EVP routing uses the Agent-advertised proxy prefix, not a hard-coded v2/v3/v4 path. The SDK keeps the track route as /api/v2/flagevaluation, but builds it under the proxy prefix discovered from the Agent. In current staging dogfooding that resolves to /evp_proxy/v4/api/v2/flagevaluation.

  • Flagevaluation reuses the existing Event Platform publisher path. This keeps delivery aligned with existing Agent discovery, headers, lifecycle, and compression controls instead of adding a Java-specific HTTP writer. Response compression is disabled for this track to match the merged Go behavior.

  • The OpenFeature hook runs inline, and under consent-on the context copy — not scalar extraction — dominates its cost. finallyAfter extracts scalar evaluation metadata, then copies the caller's evaluation context via DDEvaluator.copyPrunedContext, then does a non-blocking enqueue. Only aggregation and posting are deferred to the worker thread. This is the main correctness/performance boundary.

    The copy is bounded rather than unconditional. copyPrunedContext applies every retained-size cap inline (field count, key length, value length, list width, structure width, depth), so work is proportional to what is kept, not to what the caller supplied. It also early-returns on an empty context, so a context with no attributes costs ~15 ns rather than allocating a cycle-detection set.

    Crucially, the copy only happens under consent-on. When observeFullEvaluationData is false — the privacy-preserving default — the context is dropped on emit and never consulted by the aggregator, so the hook skips the copy entirely. Measured on FlagEvalHookHotPathBenchmark, that protected path is a flat ~7 ns regardless of context shape, while consent-on ranges from ~29 ns (empty context) to ~3.8 µs (10 nested structures × 10 fields). The javadoc previously claimed the hook did "ONLY cheap scalar extraction," which understated the consent-on cost; it has been corrected in code and quantified below.

    The copy itself is unavoidable on the consent-on path: EvaluationContext is caller-owned and mutable, so its values must be captured before the event is handed to the writer thread. Callers who enable full evaluation data and pass large or deeply nested contexts should expect low-single-digit-microsecond inline cost per evaluation.

  • The worker emits the existing batched flagevaluation contract. One flush produces a FlagEvaluationsRequest with top-level context and a flagEvaluations array; this does not use a separate /batchedflagevaluations route.

  • Aggregation keys are limited to schema-visible fields. The aggregate dimensions are flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned context. OpenFeature reason is intentionally not a hidden aggregate key because it is not serialized to the worker contract.

  • targeting_key is the single identity field for the event. The hook removes duplicate targetingKey from context.evaluation so the same identity is not encoded twice.

  • Cardinality/backpressure behavior is intentionally lossy but counted. The writer uses full-fidelity buckets first, degraded buckets without targeting key/context after cap pressure, then counted drops if both tiers or payload limits are exhausted.

  • Event time and send time are separate. Each aggregate row preserves first_evaluation and last_evaluation bounds, while the payload timestamp is the flush time.

Other Changes

  • Adds the Java EVP flagevaluation path behind DD_FLAGGING_EVALUATION_COUNTS_ENABLED while leaving the existing OTel feature_flag.evaluations hook in place.
  • Renames the existing OpenFeature metrics hook so the metrics and EVP logging hooks are easy to distinguish in review.
  • Adds tagged core metric counts for flagevaluation dropped/degraded/split telemetry.
  • Wires the writer into the feature-flagging system lifecycle with bounded queueing, periodic flush, shutdown drain, and best-effort clearing after payload encoding.
  • Adds focused unit coverage for routing, hook capture, context snapshotting, aggregation, payload encoding, writer lifecycle/posting, and system lifecycle wiring.
  • Adds two JMH benchmarks with explicitly separated scopes:
    • FlagEvalHookHotPathBenchmark (feature-flagging-api) — the evaluation-thread cost of finallyAfter across flat/nested/list context shapes, split into total inline cost (hookFinallyAfter), the bounded context copy alone (contextCopy), and the consent-off floor (hookFinallyAfterConsentOff). Adding this required enabling the me.champeau.jmh plugin on feature-flagging-api, which had no benchmark source set.
    • FlagEvaluationHotPathBenchmark (feature-flagging-lib) — writer queue mechanics and worker-thread aggregation.
    • The original single benchmark started from a pre-built flat attribute map, so it never exercised the hook's inline copy; its evalThreadCapture method is renamed writerEnqueue to stop implying it covered that cost. The hook lives in feature-flagging-api and its OpenFeature context types are not on the lib's classpath, which is why the inline cost has to be measured in a second module.
    • FlagEvaluationHotPathBenchmark now builds its events with observeFullEvaluationData=true. Under the consent-off default the aggregator drops attrs and skips canonicalContextKey entirely, so every field-count profile collapsed to the same scalar-only cost and the benchmark silently stopped measuring canonicalization.
  • Applies the repository Spotless formatter as a small follow-up commit after the narrative stack was published.
  • Adds follow-up Jacoco coverage for the feature-flagging-lib class-level coverage gate surfaced by CI.

Commit Guide

LOC is rename-aware git diff-tree -M --numstat for each commit against its parent.

SHA Changes / purpose LOC (+/-)
571938b6c7 Centralize EVP proxy endpoint construction and support the Agent-advertised proxy prefix generically. +270 / -24
4949ed7ee4 Share feature-flagging EVP publishing primitives so flagevaluation can reuse the existing transport path. +179 / -27
4aa06476b7 Add the bootstrap flagevaluation event/writer contract between OpenFeature and the agent writer. +180 / -0
ac126fb44a Rename the existing OpenFeature metrics hook so OTel metrics and EVP logging are distinct in review. +17 / -17
7a43658da6 Add the OpenFeature flagevaluation logging hook and non-blocking event capture path. +130 / -1
4ec77cae03 Cover hook capture, skip/error behavior, metadata extraction, and targeting-key de-duplication. +384 / -0
6fa7ade3e4 Snapshot OpenFeature context values at enqueue time, including nested structures/lists and duplicate scalars. +161 / -19
986a9af6be Register the flagevaluation logging hook with the Java OpenFeature provider behind the config gate. +97 / -6
f1c37aba16 Canonicalize pruned context values for deterministic aggregation keys. +194 / -0
115a407b76 Add the two-tier aggregation model for full-fidelity rows, degraded rows, and counted drops. +234 / -1
cc3b21c0c2 Cover aggregation merge keys, caps, degradation, context pruning, and constants. +183 / -0
31d7cb5116 Encode FlagEvaluationsRequest payloads, split oversized bodies, degrade oversized rows, and count drops. +270 / -0
e43f93acd0 Cover payload wire shape, split behavior, degraded rows, and error serialization. +251 / -0
17d7785420 Allow tagged core metric counts for flagevaluation drop/degradation metrics. +28 / -0
5f88cadeb2 Add writer lifecycle, bounded queue, worker thread, flush cadence, and shutdown drain. +296 / -1
597e97d10b Post encoded flagevaluation payloads through EVP and clear best-effort aggregates after encoding. +296 / -17
58df936165 Add shared test support for writer and payload tests. +212 / -0
acc6e21940 Cover writer queueing, flush, backpressure, drop metrics, shutdown, and payload posting. +304 / -0
96df04f1be Wire the flagevaluation writer into the feature-flagging system lifecycle. +22 / -0
edb264e7d8 Cover system lifecycle registration, start, and close behavior for the flagevaluation writer. +28 / -0
90ed6cb556 Add a JMH benchmark for the flagevaluation hot path. +155 / -0
c73cafdb46 Apply repository Spotless formatting after publishing the stack. +4 / -5
5b1456b3fb Add focused branch and instruction coverage for the feature-flagging-lib Jacoco gate. +465 / -2

Validation Evidence

Hot-Path Cost (JMH)

JDK 11.0.30, aarch64, 3×2s warmup / 5×1s measurement, single fork. Relative magnitudes are the point, not absolute numbers.

Evaluation-thread (inline) cost

./gradlew :products:feature-flagging:feature-flagging-api:jmh -PjmhIncludes=FlagEvalHookHotPathBenchmark

Context shape hookFinallyAfter consent-on (ns/op) contextCopy alone (ns/op) hookFinallyAfterConsentOff (ns/op) Copy share of consent-on
flat/0attrs 29.0 ± 0.9 14.7 ± 0.7 7.1 ± 0.2 51%
flat/10attrs 106.6 ± 2.4 112.1 ± 1.2 7.0 ± 0.5 ≈100%¹
flat/100attrs 1533.0 ± 64.5 1560.4 ± 80.1 6.9 ± 0.5 ≈100%¹
nested/10structs_10fields 3845.1 ± 179.4 3825.7 ± 113.1 7.3 ± 0.1 99.5%
list/10lists_10items 2146.2 ± 193.8 2083.5 ± 92.2 7.1 ± 0.6 97%

Readings:

  • The consent-off path is effectively free and flat: ~7 ns at every context shape. The hook skips the copy entirely, so cost does not scale with context size. This is the default, so most callers pay only this.
  • Under consent-on the bounded copy is essentially the entire inline cost once a context carries attributes — 97–100% across the populated shapes. Scalar extraction plus enqueue is the ~7 ns floor.
  • An empty context is cheap: ~29 ns. copyPrunedContext early-returns before allocating the cycle-detection IdentityHashMap, so there is no fixed snapshot tax on contexts with nothing to copy. An earlier revision of this PR paid ~119 ns here; that has been fixed.
  • Shape matters more than leaf count. flat/100attrs, nested/10structs_10fields, and list/10lists_10items all carry 100 leaf values, but inline cost spans 1.5–3.8 µs. Nested structures are worst (~2.5× flat) because each allocates an inner map plus a structure wrapper on top of the leaf copies.

¹ contextCopy measuring marginally above hookFinallyAfter is run-to-run variance on what is effectively the same work — the two are within ~5% of each other — not a negative-cost hook.

Writer and worker cost

./gradlew :products:feature-flagging:feature-flagging-lib:jmh -PjmhIncludes=FlagEvaluationHotPathBenchmark

Profile writerEnqueue (ns/op) workerAggregate (ns/op)
typical/100flags_50users_10fields 15.0 ± 0.7 498.2 ± 81.4
stress/10flags_1000users_250fields 15.0 ± 0.3 17317.3 ± 7574.6
scale/2500flags_500users_20fields 16.6 ± 3.4 1082.0 ± 112.2

Readings:

  • Enqueue is constant at ~15 ns and independent of context size, as intended for a non-blocking bounded-queue offer.
  • Worker aggregation scales with field count, from ~0.5 µs at 10 fields to ~17 µs at 250 fields, because canonicalizing the context key is proportional to the retained attribute count. This runs off the evaluation thread.
  • The wide error bar on the 250-field profile reflects rehashing and GC pressure in the aggregation map at that width; the order of magnitude is the signal, not the exact figure.

Local Test Gates

  • Focused Gradle gate passed after rebasing onto current origin/master (ac29db2316):
    • :communication:test
    • :products:feature-flagging:feature-flagging-api:test
    • :products:feature-flagging:feature-flagging-agent:test
    • :products:feature-flagging:feature-flagging-lib:test
    • :products:feature-flagging:feature-flagging-lib:jmhClasses
  • Covered test classes included:
    • BackendApiFactoryTest, DDAgentFeaturesDiscoveryTest
    • DDEvaluatorTest, ProviderTest, FlagEvalLoggingHookTest
    • FeatureFlaggingSystemTest
    • FeatureFlagEvpPublisherTest, FlagEvaluationAggregatorTest, FlagEvaluationPayloadsTest, FlagEvaluationWriterImplTest
  • Feature-flagging-lib coverage gate passed after the Jacoco follow-up commit:
    • ./gradlew :products:feature-flagging:feature-flagging-lib:test :products:feature-flagging:feature-flagging-lib:jacocoTestReport :products:feature-flagging:feature-flagging-lib:jacocoTestCoverageVerification
  • Formatting/lint checks after the Spotless and coverage follow-up commits:
    • ./gradlew spotlessApply
    • ./gradlew spotlessCheck
    • ./gradlew :products:feature-flagging:feature-flagging-lib:spotlessCheck
    • ./gradlew :communication:forbiddenApis :dd-trace-core:forbiddenApis :internal-api:forbiddenApis :telemetry:forbiddenApis :products:feature-flagging:feature-flagging-api:forbiddenApis :products:feature-flagging:feature-flagging-agent:forbiddenApis :products:feature-flagging:feature-flagging-lib:forbiddenApis
  • Stack hygiene:
    • git diff --check passed.
    • All published commits verified with good git signatures.

Dogfooding App

  • Rebuilt ffe-dogfooding Java artifacts from this local dd-trace-java stack with scripts/prepare-local-java.sh.
  • Restarted dogfooding with local dd-openfeature and dd-java-agent artifacts plus the real backend EVP path.
  • Java app health reached PROVIDER_READY.
  • Evaluated ffe-dogfooding-string-flag through the Java dogfooding app 15 times total: 5 evaluations for each targeting key:
    • java-restack4-20260702T042247Z-alpha
    • java-restack4-20260702T042247Z-bravo
    • java-restack4-20260702T042247Z-charlie
  • App-side result: all 15 evaluations returned variant_1, allocation allocation-override-392dd7c149f8, service java, and evaluation reason STATIC.
  • App logs showed two successful EVP posts to the Agent-advertised route http://datadog-agent:8126/evp_proxy/v4/api/v2/flagevaluation, both returning 202.

Staging End-To-End

  • Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
  • Retriever staging query against eventplatform.system.track(TRACK => 'flagevaluation') returned 3 aggregated rows for the exact targeting keys above.
  • Each row had:
    • flag.key=ffe-dogfooding-string-flag
    • variant.key=variant_1
    • allocation.key=allocation-override-392dd7c149f8
    • evaluation_count=5
  • This proves SDK aggregation/batching for the final local tree: 15 app evaluations became 3 backend flagevaluation rows.

System Tests

  • Companion draft PR: Enable EVP flagevaluation system tests for Java system-tests#7185
  • Local manifest-enabled Java EVP flagevaluation system tests passed against PR head 6b7aa4273d:
    • TEST_LIBRARY=java ./run.sh +v FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_evp.py
    • Result: 8 passed in 80.08s (Library: java@1.64.0-SNAPSHOT+6b7aa4273d, Weblog variant: spring-boot).

Integration Addendum

JAVA-01 is the direct PR base and is preserved as an ancestor through signed merge commit d5b90bad24. FeatureFlaggingSystem now selects the configured source first, then starts exposure and aggregate-evaluation writers independently for agentless, Remote Configuration, and reserved offline source modes.

Validation Evidence

  • Feature-flagging API, agent, and lib tests plus focused Spotless checks: pass.
  • Local artifacts: dd-java-agent, dd-trace-api, and dd-openfeature: pass.
  • System tests: agentless 6 passed; existing RC aggregate EVP 1 passed.
  • Dogfooding with authenticated default-agentless UFC: exposure 10, aggregate EVP 1, OTLP 1.

CI Packaging Decision

Any custom Java system-test build must install all three artifacts. Supplying only dd-java-agent and dd-trace-api leaves the weblog on the published dd-openfeature implementation and does not exercise the PR head.

@datadog-datadog-prod-us1-2

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.81 s 14.73 s [-0.3%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 13.58 s 13.73 s [-1.9%; -0.3%] (maybe better)
startup:petclinic:appsec:Agent 17.46 s 17.29 s [-0.0%; +2.0%] (no difference)
startup:petclinic:iast:Agent 16.65 s 17.53 s [-9.1%; -0.9%] (maybe better)
startup:petclinic:profiling:Agent 17.39 s 17.41 s [-1.3%; +1.1%] (no difference)
startup:petclinic:sca:Agent 17.36 s 17.09 s [+0.5%; +2.6%] (maybe worse)
startup:petclinic:tracing:Agent 16.62 s 16.20 s [-1.7%; +6.8%] (no difference)

Commit: ac5c52b1 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@leoromanovsky leoromanovsky changed the title [FFL-2446] dd-trace-java: emit EVP flagevaluation (Phase 2 fan-out) feat(openfeature): emit server-side EVP flagevaluation Jun 14, 2026
@leoromanovsky
leoromanovsky marked this pull request as ready for review June 23, 2026 00:23
@leoromanovsky
leoromanovsky requested review from a team as code owners June 23, 2026 00:23
@leoromanovsky
leoromanovsky requested review from PerfectSlayer, bric3, dd-oleksii and typotter and removed request for a team June 23, 2026 00:23
@dd-octo-sts

dd-octo-sts Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@leoromanovsky leoromanovsky added type: feature Enhancements and improvements comp: openfeature OpenFeature labels Jun 23, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d4244f8ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread communication/src/main/java/datadog/communication/BackendApiFactory.java Outdated
@leoromanovsky
leoromanovsky requested a review from a team as a code owner June 23, 2026 19:41
@leoromanovsky
leoromanovsky force-pushed the leo.romanovsky/ffl-2446-evp-flagevaluation-java branch from 83ac4c4 to 81ed6f1 Compare July 2, 2026 00:02
@leoromanovsky
leoromanovsky marked this pull request as draft July 2, 2026 03:15
vjfridge and others added 4 commits August 7, 2026 20:30
…t logging

At shutdown there is nowhere to surface the error usefully, and each
writer already logs internally on failure. Debug-logging from the
catch blocks was misleading noise. True swallow (Exception ignored)
matches the intent of the original suggestion.

Co-Authored-By: Claude <noreply@anthropic.com>
…ingSystem.stop()

Replaces four identical try/catch blocks with a single closeQuietly
helper. SpanEnrichmentWriter gains AutoCloseable (it already had a
close() method) so all four resources share the same call site.

Co-Authored-By: Claude <noreply@anthropic.com>
…tionAggregator

EVAL_SCALE_ prefix was ambiguous — it read as a runtime scale factor
rather than design-time sizing assumptions. Split into two named groups:
- EXPECTED_* for the design assumptions (flag count, users per flag, etc.)
- *_SIZING_BASIS for the derived intermediate values
- Inline comments on GLOBAL_CAP and DEGRADED_CAP explain they are the
  nearest powers of two above the respective sizing bases.

Co-Authored-By: Claude <noreply@anthropic.com>
The multi-pass loop with Thread.yield() between passes was a heuristic
for catching producers mid-enqueue during shutdown. A single poll loop
already drains everything in the queue at that point. For events that
race past the drain, close() sweeps the queue after joining the worker
and counts any remainder as an observable drop — making extra passes
redundant. Removed SHUTDOWN_DRAIN_PASSES constant and the loop.

Co-Authored-By: Claude <noreply@anthropic.com>
vjfridge and others added 5 commits August 10, 2026 11:05
The lazy Supplier overload and contextAttributes() accessor were removed
in 29b974d, breaking compileTestJava. Drop the two tests that covered
the deleted lazy path and remove the contextAttributes() assertions from
the remaining tests.

Co-Authored-By: Claude <noreply@anthropic.com>
…verload

The lazy Supplier constructor was removed in 29b974d; pass attrs directly.
Also rename nextLazyEvent -> nextEvent in FlagEvaluationHotPathBenchmark.

Co-Authored-By: Claude <noreply@anthropic.com>
Commit 29b974d removed the lazy-supplier path and took two tests with
it, dropping FlagEvaluationWriterImpl branch/instruction coverage below
the 90% Jacoco threshold on Java 8. Add two replacement tests:
- scoConstructorCreatesUsableWriter: exercises the public SCO constructor
- countContextTruncatedAccumulatesPerReason: exercises countContextTruncated

Co-Authored-By: Claude <noreply@anthropic.com>
The pre-queue guard methods were only exercised through mocks in the hook
test, leaving FlagEvaluationWriterImpl branch coverage at 0.844 and failing
jacocoTestCoverageVerification (threshold 0.9). Add a direct test that
observes both branches of hasCapacityForEnqueue and the countPreQueueOverflow
counter surfacing as a queue_overflow drop metric.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@sabrenner sabrenner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llmobs span mapper changes lgtm - i believe this follows our node.js & python limits as well

…ions events (#12042)

* Parse observeFullEvaluationData and hash targeting_key in flagevaluations events

Adds the top-level observeFullEvaluationData boolean to the UFC model,
plumbs it through to the EVP flagevaluation event serializer, and gates
PII handling on it: when the flag is absent/false the targeting key is
SHA-256 hashed (sha256_<hex>) and the raw evaluation context is omitted
from the wire; when true the raw targeting key and context are emitted.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Extract hashed targeting key prefix into a named constant

Replace the inline "sha256_" literal with a documented
HASHED_TARGETING_KEY_PREFIX constant describing the cross-SDK wire
contract for privacy-preserving hashed targeting keys.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Test observeFullEvaluationData parsing edge cases

Parameterize the true/false config-parsing assertions with @valuesource
and add a test locking in the fail-closed behaviour for an explicit JSON
null: malformed config is rejected so full evaluation data is never
observed off the back of it.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Capture observeFullEvaluationData per bucket at aggregation time

The flush-time read of FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled()
was a TOCTOU bug: CURRENT_CONFIG could be overwritten by a later RC update
between when an evaluation happened and when the batch flushed, so events
could be emitted under the wrong environment's consent (the system test
observed a targeting key hashed even though the active UFC said
observeFullEvaluationData=true).

Capture consent when the evaluation is folded into its EvalBucket instead.
On merge the value is folded with AND, so any no-consent evaluation in a
bucket's lifetime sinks the whole bucket to hashed/omitted (fail-closed).
buildEventList now reads bucket.observeFullEvaluationData rather than the
gateway. The gateway accessor is retained; it is read at aggregation time.

Adds a writer-level regression guard (a bucket aggregated under consent-off
stays hashed even if the gateway later reports consent-on) plus aggregator
fold tests, and an end-to-end parse->dispatch->flush test.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Capture observeFullEvaluationData consent at evaluation time

Snapshot the PII consent flag on the evaluation thread (in the OpenFeature hook) and carry it on FlagEvalEvent, instead of reading the gateway when the event is aggregated/flushed. This pins the hashed-vs-raw decision to the configuration active at evaluation time, closing a one-directional leak window where a later Remote Config update could retroactively apply a different environment's consent to already-collected evaluations.

Aggregation and flush now read event.observeFullEvaluationData and never consult the gateway; the AND-fold across a bucket's evaluations is unchanged (any no-consent evaluation sinks the bucket to hashed/omitted).

Environment: Datadog workspace

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Bind observeFullEvaluationData consent to the evaluator's configuration

Address PR #12042 review feedback (Codex P1, leoromanovsky, dd-oleksii): the
FlagEvalLoggingHook was reading observeFullEvaluationData from
FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() at hook-fire time,
which races against a Remote Config swap of CURRENT_CONFIG that happens after
DDEvaluator.evaluate() captured its own ServerConfiguration reference. That
race can retroactively mark an evaluation performed without consent as
consented and leak the raw targeting key / context.

DDEvaluator now stamps the boolean directly from the ServerConfiguration it
used, onto every ProviderEvaluation via ImmutableMetadata under key
"dd.observe_full_evaluation_data". The hook reads consent from that metadata
and no longer queries the gateway. Missing metadata (PROVIDER_NOT_READY or a
non-DD provider) → false, the privacy-preserving default.

The gateway's isObserveFullEvaluationDataEnabled() accessor is removed since
its only real caller was the hook and re-adding it would re-open the race.

Adds regression tests: hook honours consent metadata (true/false/absent) and
ignores a gateway value that disagrees; evaluator stamps the correct boolean
on the FLAG_NOT_FOUND path and omits metadata when it holds no config.

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Pass the boolean, not the ServerConfiguration, into error/resolveVariant

Follow-up to the previous commit: the private error() and resolveVariant()
helpers only ever read one field off the ServerConfiguration
(observeFullEvaluationData), so pass the boolean directly instead of the whole
config. Keeps the internal API narrow and removes the incidental coupling
these helpers had to the UFC.

While here, PROVIDER_NOT_READY now stamps consent as the privacy-preserving
false rather than omitting the metadata. Same on-the-wire outcome the hook
would have produced, but the invariant "every DD-produced evaluation carries
dd.observe_full_evaluation_data" is now unconditional, which is easier to
reason about. The two error() overloads collapse to one (the (String) null
casts at call sites disappear along with them).

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Drop the dd. prefix on the evaluation-metadata consent key

The key is only ever read by FlagEvalLoggingHook one line later — it never
lands on the wire, so it doesn't need the "dd." namespacing that
"dd.eval.timestamp_ms" has (that key is re-emitted onto spans).

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Trim verbose comments around the observeFullEvaluationData plumbing

The race-vs-CURRENT_CONFIG backstory is captured in the previous commits'
messages; the code only needs the forward-looking invariants (metadata is
source of truth, missing key = false, DD-produced evaluations always stamp).

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Skip evaluation context when aggregating consent-off evaluations

Address PR #12042 review from leoromanovsky (escalated Codex P2 → P1): on the
protected path (observeFullEvaluationData=false) the serializer drops the
evaluation context, but the aggregator was still running it through
pruneContext + canonicalContextKey and keying every full-tier bucket on it. A
high-cardinality field on the evaluation context (request_id, timestamp,
correlation id) would fragment buckets that emit byte-identical wire rows,
blow out PER_FLAG_CAP (10k) inside one flush window, and force subsequent
evaluations into the degraded tier — which drops the targeting key entirely.

On the protected path aggregate() now uses ctxKey="" and stores
prunedAttrs=null, so different contexts for the same subject collapse into one
bucket. The targeting key stays in the aggregation identity, so different
subjects still hash to different buckets. The consent-on path is unchanged.

Regression tests: protected path collapses differing contexts for one subject;
protected path still separates distinct subjects; full path still splits on
context. Existing tests that exercise pruneContext / context-differentiation
were updated to use consent=on (that's the code path they actually cover).

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Skip evaluation-context capture on the hook hot path when consent is off

Companion to the aggregator fix: with observeFullEvaluationData=false the
evaluation context is dropped on emit and no longer influences aggregation,
so there is no reason to snapshot it on the evaluation thread. The hook now
branches on consent up front — the protected path enqueues an event with an
empty materialized attrs map (no map copy of the OpenFeature context, no
Supplier<Map> allocation, no lambda instance), while the consent-on path is
unchanged.

Grep confirms the only production consumer of FlagEvalEvent.contextAttributes
/ FlagEvalEvent.attrs is FlagEvaluationAggregator.aggregate, which already
skips them on the protected path.

Regression test: mutating the EvaluationContext after finallyAfter returns
still yields empty attrs on the enqueued event — proves the hook never
snapshotted it. Two existing tests that exercise the snapshot mechanism were
switched to pass consent-on metadata (that's the code path they cover).

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Include observeFullEvaluationData in the aggregation bucket key

Bucket keys should cover every dimension the emitter will branch on. The
serializer branches on observeFullEvaluationData (hashes the targeting key
and drops the context when off), so two evaluations that differ only in
consent produce different wire rows and must not share a bucket.

Before this change they could: same subject, same flag, same empty context
would land under the same FullKey regardless of consent, and the AND-fold
would silently downgrade a consent-on evaluation to the protected wire shape
because a nearby consent-off event merged into its bucket first. No PII leak
(fail-closed direction), but arrival-order-dependent semantics and a lost
raw-context row.

Add observeFullEvaluationData to FullKey / DegradedKey (equals + hashCode).
The AND-fold on bucket.observeFullEvaluationData stays as defensive belt-
and-suspenders; every event merging into a bucket now carries the matching
consent value by construction.

Regression test: two events identical except for consent land in two full-
tier buckets, one consent-on and one consent-off. Updated the previous
"fold to false on mixed consent" test to reflect the new invariant.

Generated with Claude Code

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add consent metadata to ProviderTest flag-eval-logging hook route test

The test asserts that context attributes flow through the logging hook,
but the mock metadata omitted the observe-full-evaluation-data flag, so
the hook took the privacy-preserving path and dropped context.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Redact error messages when observeFullEvaluationData is off

Exception messages from the evaluator's outer catch blocks
(NumberFormatException, generic Exception) can echo raw evaluation-context
values verbatim — for example a GT rule on "id" with a PII-shaped targeting
key produced error.message="For input string: \"jane.doe@...\"" on the wire
regardless of consent, defeating the PR's own PII guard. Drop the message
at DDEvaluator.error() when consent is off, and add a hook-layer fallback
that substitutes ErrorCode.name() so operators keep a stable signal
(e.g. "TYPE_MISMATCH") even when a third-party provider hands us a raw
message.

Generated with Claude Code
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Exercise every consent-stamp code path in DDEvaluatorTest

The only observeFullEvaluationData assertions were on error paths
(FLAG_NOT_FOUND, PROVIDER_NOT_READY), leaving the success-path stamp in
resolveVariant and the DISABLED/DEFAULT stamps in consentMetadata
uncovered — line 448 could be deleted or hardcoded to either value and
every existing test would still pass. Add symmetric consent-on/consent-off
tests for each of resolveVariant, DISABLED, and DEFAULT so any mutation
(delete / hardcode true / hardcode false) flips at least one assertion.
Rename the previously misleading …OnSuccess test to reflect what it
actually exercises (FLAG_NOT_FOUND error via error()).

Generated with Claude Code
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Drop consent from DegradedKey to reclaim effective DEGRADED_CAP

Two degraded buckets differing only in observeFullEvaluationData emit
byte-identical wire JSON — the degraded serializer (fromBucket with
isFullTier=false) drops the targeting key and context regardless of
consent — so the consent dimension in DegradedKey halved effective
DEGRADED_CAP for zero wire fidelity gain. FullKey correctly keeps
consent (the full-tier serializer branches on it for raw-vs-hashed
targeting key and context inclusion).

Mixed-consent events now merge into one degraded bucket. The AND-fold
on EvalBucket.observeFullEvaluationData still runs and collapses to
false whenever any consent-off event lands in a mixed bucket; benign
because the value has no downstream effect for degraded rows.

Generated with Claude Code
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Tolerate malformed observeFullEvaluationData in UFC parse

Before this change ServerConfiguration.observeFullEvaluationData was a
primitive boolean — Moshi's reflective adapter rejected the entire UFC
whenever the JSON value was null or wrong-typed. Agentless swallows the
IOException at DEBUG, so a pod starting after a malformed message had
no last-known-good, stranded every flag on PROVIDER_NOT_READY, and
served defaults forever. Fail-closed on privacy shouldn't cascade into
fail-closed on availability.

Box the field to Boolean so null tolerates naturally, register a
LenientBooleanAdapter that maps wrong-typed values to null as well,
and read via Boolean.TRUE.equals(...) at the DDEvaluator so null falls
to the privacy-preserving default. The lenient adapter only intercepts
Boolean (not primitive boolean), so mandatory fields like Flag.enabled
keep their strict parse; the only other Boolean it touches is
Allocation.doLog, which is already read as `!= null && doLog`.

Reversed the earlier RejectsExplicitNull test — it had locked in the
buggy behaviour — into a family of tolerance tests for null / stringified
/ numeric. Added a DDEvaluator test that a config with a null consent
field evaluates without NPE and stamps the privacy-preserving default.

Generated with Claude Code
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Set observeFullEvaluationData=true for the NaN-poison flush test

The consent-off short-circuit in FlagEvaluationEvent.fromBucket drops the
raw context before Moshi encodes it, so a NaN in the attrs never reaches
the encoder and the flush succeeds. That defeated the intent of
encodeFailureClearsAggregatorSoLaterFlushesRecover, which must observe
a real encode failure to prove the aggregator is cleared.

Co-Authored-By: Claude <noreply@anthropic.com>

* Cover LenientBooleanAdapter read-only and qualifier paths

The per-class JaCoCo gate (0.9 minimum, gradle/jacoco.gradle) failed on
the new adapter: toJson was never invoked (20/25 instructions) and the
factory's !annotations.isEmpty() short-circuit never evaluated true
(3/4 branches). Neither path is reachable through the parse-driven
tests in JsonApiUfcResponseParserTest.

Mirror the tests the sibling FlagMapAdapter and DateAdapter already
have. The primitive-boolean assertion documents the guard that keeps
this leniency off mandatory fields like Flag.enabled.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@vjfridge

Copy link
Copy Markdown
Contributor

📢 FYI I merged #12042 into this branch so we have PII protected when we merge to master!

…ng/canonicalization

Address review feedback from @AlexeyKuznetsov-DD on the FlagEvaluationAggregator:

- Replace String.format("%08x", ...) in appendLengthDelimited with a
  StringBuilder-based zero-padded hex writer. Format on the per-context-field
  path allocates a Formatter and boxes the length argument on every call.
- Rewrite FullKey.hashCode and DegradedKey.hashCode with HashingUtils.hash /
  addToHash. Objects.hash allocates an Object[] per call and boxes the boolean
  dimensions; both keys are looked up on every event dispatch.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment on lines 71 to 76
try {
initializeSystem(sco, config);
} catch (final RuntimeException | Error e) {
STARTED = false;
throw e;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code looks like a duplicated (2 same places).
And also probably is missing of safe closing of exposureWriter and configService.
It is like partially initialized state did not rolled back.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on both points, thank you. Fixed in 2c0d326.

Duplication: the identical try { initializeSystem } catch { STARTED = false; throw } block in start() and activateAgentless() is now a single private helper, initializeOrRollBack.

Partially initialized state: you are right that the rollback was incomplete. The old handler only cleared STARTED. If initializeSystem threw after initialize() had already published CONFIG_SERVICE and EXPOSURE_WRITER (for example from evalWriter.start() or SpanEnrichmentWriter.init()), those two stayed open and unreachable, and the gateway kept its enqueue flag set. The helper now calls stop() on failure, which closes every published resource, removes any activation listener, clears the gateway flags, and clears STARTED. stop() is synchronized on the same class monitor the caller already holds, so the reentrant call is safe, and it was already idempotent, so it tolerates being called when only part of the state exists.

While there I fixed one more instance of the same problem: FLAG_EVAL_WRITER was assigned after evalWriter.start(), so a writer whose start() failed was unreachable and no rollback could close it. The assignment now comes first.

The pre-existing inner rollback in initialize() for the config-service and exposure-writer pair is unchanged and still correct. The new outer rollback covers the wider window that opens after initialize() returns.

Added failedStartRollsBackPartiallyInitializedState to cover it: a failed start must leave no pending activation listener, no gateway writer, enqueue disabled, and a restartable subsystem.

The failure handler only cleared STARTED. If initializeSystem threw after
initialize() had published CONFIG_SERVICE and EXPOSURE_WRITER, those stayed
open and unreachable, and the gateway kept its enqueue flag.

Route both start paths through one initializeOrRollBack helper that calls
stop() on failure, which closes every published resource, removes any
activation listener, and clears the gateway flags. This also removes the
duplicated try/catch shared by start() and activateAgentless().

Also publish FLAG_EVAL_WRITER before evalWriter.start(), so a writer whose
start fails is still reachable by the rollback.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
vjfridge and others added 2 commits August 11, 2026 15:30
FlagEvaluationHotPathBenchmark built its events with the convenience
constructor, which defaults observeFullEvaluationData to false. Under
consent-off the aggregator drops attrs and skips canonicalContextKey,
so every field-count profile measured the same scalar-only work: 10
fields and 250 fields both landed near 30 ns. Pass consent true so the
benchmark exercises canonicalization again. The profiles now spread
from 498 ns at 10 fields to 17.3 us at 250 fields.

Add FlagEvalHookHotPathBenchmark in feature-flagging-api to cover the
evaluation-thread cost the lib benchmark cannot reach. It splits total
inline cost, the bounded context copy alone, and the consent-off floor
across flat, nested, and list context shapes. This required enabling
the me.champeau.jmh plugin on the module, which had no benchmark
source set.

Also refresh the lib benchmark javadoc, which still described an
attribute-supplier API that no longer exists and claimed the hook cost
was unmeasured anywhere.

Co-Authored-By: Claude <noreply@anthropic.com>
@vjfridge

Copy link
Copy Markdown
Contributor

System-tests validation against this branch (local run)

Ran the full FFE system-tests surface against this branch to confirm the EVP flagevaluation work passes end-to-end and doesn't regress existing Java FFE behavior.

Tracer under test: 1.65.0-SNAPSHOT~a1860df8d0 (this branch's HEAD, with master already merged in)
system-tests: PR #7185 (un-skips tests/ffe/test_flag_eval_evp.py for Java) with latest main merged in.

How it was built

FFE needs three jars in binaries/, not just the agent — the spring-boot weblog's pom.xml depends on a dd-openfeature artifact, which is where the OpenFeature provider lives. Building only dd-java-agent.jar would leave the provider coming from a released artifact and silently under-test these changes.

./gradlew :dd-java-agent:shadowJar \
          :products:feature-flagging:feature-flagging-api:jar \
          :dd-trace-api:jar
# copy ONLY the plain jars into system-tests/binaries/
# (install_ddtrace.sh hard-fails if its glob matches >1, and gradle
#  also emits -sources / -javadoc / -jmh variants)

./build.sh -i weblog -l java -w spring-boot
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION
./run.sh PARAMETRIC --library java tests/parametric/test_ffe/

Results

End-to-end (FEATURE_FLAGGING_AND_EXPERIMENTATION, spring-boot) — 47 passed, 1 xfailed, 0 failed in 5m51s

File Result
tests/ffe/test_flag_eval_evp.py 11/11 passed (newly enabled by #7185)
tests/ffe/test_exposures.py 10 passed, 1 xfail
tests/ffe/test_flag_eval_metrics.py 17 passed
tests/ffe/test_dynamic_evaluation.py 7 passed

All 11 EVP classes were confirmed individually from reportJunit.xml (not just counting pytest dots) to have actually run rather than skipped:

Test_FFE_EVP_Flagevaluation_Basic                        PASSED
Test_FFE_EVP_Flagevaluation_Count                        PASSED
Test_FFE_EVP_Flagevaluation_Context_Bounds               PASSED
Test_FFE_EVP_Flagevaluation_Runtime_Default              PASSED
Test_FFE_EVP_Flagevaluation_Load_Aggregation             PASSED
Test_FFE_EVP_Flagevaluation_Burst_Aggregation            PASSED
Test_FFE_EVP_Flagevaluation_High_Cardinality_Aggregation PASSED
Test_FFE_EVP_Flagevaluation_Degradation                  PASSED
Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed  PASSED
Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed   PASSED
Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed  PASSED

This covers the PII targeting-key hashing vectors, aggregation/counting, context bounds, runtime defaults, and the degradation/overflow path.

Parametric (tests/parametric/test_ffe/) — 40 passed, 12 xfailed, 19 xpassed, 0 failed in 76s. No regressions to existing Java FFE functionality.

Notes on the non-failures

  • 1 xfail (e2e): Test_FFE_EXP_5_Missing_Targeting_Keybug (FFL-1729). Pre-existing manifest declaration, unrelated to this branch.
  • 19 xpass (parametric): all in test_configuration_sources.py, which the Java manifest gates at v1.65.0. A 1.65.0-SNAPSHOT build sorts below the 1.65.0 release, so these are marked expected-to-fail and then pass. An artifact of testing a pre-release build; resolves once 1.65.0 ships. Not caused by this branch and no manifest change made for it.

Jar provenance

Verified the tests actually exercised this branch's code rather than a released artifact — SHA-256 identical across gradle output → binaries/ → inside the built weblog image:

Jar SHA-256 (all 3 locations)
dd-openfeature aa74d8025756b4e64464a1e2ffd3863d1848a6dc666cf3a29f14570fe241d1f9
dd-java-agent 615e4999ba2fa2bf5dda499965c7473fc1d88bac0cb2cce15bd1066b9947374f

Additional confirmation: the weblog image contains BOOT-INF/lib/dd-openfeature-9999.jar (the 9999 sentinel is what install_ddtrace.sh assigns to a custom jar, so the GitHub-release download path did not run), and the running JVM logged its own version during the test window:

DATADOG TRACER CONFIGURATION {"version":"1.65.0-SNAPSHOT~a1860df8d0", ...}

🤖 Generated with Claude Code

@vjfridge

Copy link
Copy Markdown
Contributor

Manual validation against staging (ffe-dogfooding, Java SDK)

Ran this branch end-to-end through the ffe-dogfooding stack against the staging org (dd.datad0g.com, DD_ENV=staging) with live Remote Config. Tested commit a1860df8d0 — matches this PR's head, no local modifications.

Setup

DD_TRACE_JAVA_PATH=~/dd/dd-trace-java scripts/prepare-local-java.sh
DD_TRACE_JAVA_PATH=~/dd/dd-trace-java docker compose \
  -f docker-compose.yml -f local/docker-compose.java.yml \
  up --build -d mock-intake otlp-intake datadog-agent app-java evaluator

Locally built dd-java-agent-1.65.0-SNAPSHOT~a1860df8d0 + dd-openfeature, config delivered over Agent Remote Configuration, EVP events captured at mock-intake via the agent's EVP proxy.

Results

Provider + evaluations — 170 evaluations, 100% success, 0 failures.

  • PROVIDER_READY in 75 ms; live PROVIDER_CONFIGURATION_CHANGED events observed as RC pushed updates.
  • All 6 dogfooding flags returned real staging variants, not defaults. ffe-dogfooding-string-flag produced a genuine 3-way split (control 9 / variant_1 13 / variant_2 7, reason SPLIT), confirming real allocation rather than fallback values.
  • Error paths (FLAG_NOT_FOUND, TYPE_MISMATCH) returned defaults with variant.key null, per the FFL-2969 contract.

flagevaluation EVP events land on /api/v2/flagevaluation with correct aggregation fields (evaluation_count, first_evaluation, last_evaluation) and real allocation keys.

observeFullEvaluationData — correct in both directions.

Worth noting for anyone reproducing: observe_full_evaluation_data is an environment-level setting (/api/v2/feature-flags/environments/<env-id>), not per-flag — the per-flag feature_flag_environments[] entries just mirror it. Flipping it affects every flag in the environment.

consent ON consent OFF
targeting_key raw (user-0gu401kuar6q) sha256_a320ad9b…
context full attributes present field absent entirely
mock-intake verdict full protected

Consent ON:

{
  "targeting_key": "user-0gu401kuar6q",
  "context": {"evaluation": {"country": "US", "plan": "pro", "version": "1.0.0"}},
  "flag": {"key": "ffe-dogfooding-float-flag"},
  "variant": {"key": "-3.14"},
  "allocation": {"key": "allocation-override-b84f868b6ff5"},
  "evaluation_count": 1
}

Consent OFF:

{
  "targeting_key": "sha256_a320ad9bdc04de267751bb3742b3814de63aea1b41c3bbc62eca83c09f59ad46",
  "flag": {"key": "ffe-dogfooding-string-flag"},
  "variant": {"key": "control"},
  "allocation": {"key": "3baabb3c-2471-4ef8-b04e-1ab20790abe9"},
  "evaluation_count": 1
}

Audited all 74 events captured under consent-OFF:

  • 100% classified protected
  • 0 raw (unhashed) targeting keys
  • 0 events carrying a context object
  • Raw string scan of the full payload set for the subject ID, plan, country, pro, USno matches
  • Verified the digest is a genuine unsalted SHA-256 of the raw targeting key (sha256("consent-off-probe") matched exactly) — real hashing, not truncation or a placeholder

The functional payload survives redaction in both modes: allocation key, variant, and aggregation counters are intact under consent-OFF, so only subject-identifying data is dropped.

The toggle propagated live in both directions without an app restart — flipping consent off moved the verdict to protected, and reverting moved it back to full.

Staging was restored to its original state afterward and verified byte-for-byte against a pre-change GET.

close() interrupts the worker thread to break it out of
queue.poll(100ms). The worker then falls into the finally block in run(),
which calls drainAndFlush() to do the final flush that close() exists to
guarantee. The interrupt flag is still set at that point.

That flush does socket I/O, and OkHttp fails fast on a thread whose
interrupt flag is set. The resulting IOException is swallowed by the
broad catch in flush(), whose finally then calls aggregator.clear(). So
every aggregated evaluation in the final flush window was discarded on
every clean shutdown, and the loss was invisible: it surfaced as an error
log rather than a drop metric.

close()'s post-join sweep does not cover this. The sweep counts events
still sitting in the queue. These rows had already been drained out of
the queue into the aggregator before flush() threw, so the sweep cannot
see them.

Save and clear the interrupt flag around drainAndFlush(), then restore it
so the thread still exits with the correct interrupt status.

The bug needed both halves to appear, and neither is wrong alone: the
interrupt came from 5c6cb2b ("Make flagevaluation enqueue lock-free"),
which needs it to wake the worker now that the lock no longer orders
offers against the drain; the finally-block drain came from 5f88cad.

The existing closeDrainsAndFinalFlushesQueuedEvents test passes against
the bug because a mocked publisher ignores the interrupt flag - only a
real socket reacts to it. The new test therefore asserts on the flag
state at the moment post() is invoked. Verified it fails without the
production change: "expected: <false> but was: <true>".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vjfridge
vjfridge added this pull request to the merge queue Aug 12, 2026
@dd-octo-sts

dd-octo-sts Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 12, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-12 14:21:31 UTC ℹ️ Start processing command /merge


2026-08-12 14:21:36 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).


2026-08-12 15:27:46 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 12, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 95065ee into master Aug 12, 2026
594 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffl-2446-evp-flagevaluation-java branch August 12, 2026 15:27
@github-actions github-actions Bot added this to the 1.66.0 milestone Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants