Add native secret-store config resolution

[ChristianPavilonis]

Summary

• Store references to static credentials in Trusted Server configuration, then resolve those references from the platform's secret store after the signed configuration blob passes integrity checks.
• Resolve publisher, Edge Cookie partner, handler, Tinybird, DataDome, and S3 credentials while loading typed application configuration. Runtime code receives redacted values and no longer reads these static credentials during requests.
• Keep trusted_server_secrets as the logical store name while allowing adapters to map it to a physical store such as Fastly's ts_secrets. Missing or invalid secrets fail configuration loading without exposing their values.
• Accept the old feature-specific secret_store selectors for one release, warn that they are ignored, and omit them when serializing configuration.
• Make Edge Cookie partner api_token references optional. Partners without one remain available for source-domain lookup, bidstream EIDs, and outbound pull sync, but cannot authenticate to the inbound identify or batch-sync APIs. ts_pull_token remains required only when pull sync is enabled.

This fixes the deployment failure where a valid secret existed in Fastly but Trusted Server opened the logical store name instead of the mapped physical store.

Changes

File	Change
.env.dev	Clarify that the file contains non-secret development overlays and point local users to the config blob and secret-store setup.
.env.example	Document logical-to-physical secret-store mapping and replace plaintext secret examples with platform secret-store guidance.
Cargo.toml	Pin the EdgeZero revision that supports optional secret paths and persisted Fastly store mappings.
Cargo.lock	Record the updated EdgeZero dependency graph.
crates/trusted-server-adapter-axum/src/app.rs	Pass the Axum secret-store adapter into typed settings loading.
crates/trusted-server-adapter-cloudflare/src/app.rs	Resolve configuration references through the Cloudflare Worker environment during startup.
crates/trusted-server-adapter-cloudflare/src/lib.rs	Make the Worker environment available to startup configuration loading.
crates/trusted-server-adapter-cloudflare/src/platform.rs	Expose the Cloudflare secret-store adapter within the crate for configuration resolution.
crates/trusted-server-adapter-cloudflare/wrangler.ci.toml	Add fictional local secret bindings used by Cloudflare integration tests.
crates/trusted-server-adapter-cloudflare/wrangler.toml	Document how operators provision Worker secrets referenced by application configuration.
crates/trusted-server-adapter-fastly/src/app.rs	Load Fastly runtime store mappings, resolve typed secrets at startup and reload, and cover mapped-store behavior with tests.
crates/trusted-server-adapter-fastly/src/main.rs	Build the Fastly application with the runtime environment mapping used by EdgeZero.
crates/trusted-server-adapter-fastly/src/tinybird.rs	Use the Tinybird token resolved during configuration loading instead of reading a secret store during each request.
crates/trusted-server-adapter-spin/spin.toml	Declare Spin secret variables for application-config references.
crates/trusted-server-adapter-spin/src/app.rs	Pass the Spin secret store into startup configuration loading.
crates/trusted-server-adapter-spin/src/platform.rs	Add the Spin adapter used to resolve typed application secrets.
crates/trusted-server-core/src/config.rs	Mark secret-bearing fields, make partner API-token references optional, add conditional requirements, support the deserialize-only selector bridge, and split deploy-time structure checks from post-resolution validation.
crates/trusted-server-core/src/config_payload.rs	Verify blob integrity before resolving references and add fail-closed tests for missing, malformed, inactive, and optional secrets, including partners without API tokens.
crates/trusted-server-core/src/ec/auth.rs	Keep inbound bearer authentication fail-closed when a configured partner has no API token.
crates/trusted-server-core/src/ec/registry.rs	Register every partner by source domain while hashing and indexing only configured API tokens; validate partner structure before deployment and defer token-value checks until references have been resolved.
crates/trusted-server-core/src/integrations/datadome.rs	Load DataDome credentials into redacted runtime settings and ignore the old store selector.
crates/trusted-server-core/src/integrations/datadome/protection.rs	Use the resolved DataDome key without a request-time secret-store lookup while retaining the configuration-gated test bypass.
crates/trusted-server-core/src/lib.rs	Export the secret-resolution module.
crates/trusted-server-core/src/proxy.rs	Use resolved publisher and S3 credentials and remove feature-specific runtime secret-store reads.
crates/trusted-server-core/src/publisher.rs	Update publisher tests for DataDome's typed secret reference.
crates/trusted-server-core/src/secret_resolution.rs	Add recursive typed resolution for nested objects, arrays, optional containers, and redacted errors.
crates/trusted-server-core/src/settings.rs	Separate reference-bearing application configuration from resolved runtime settings, make partner API tokens optional, and sanitize validation failures.
crates/trusted-server-core/src/settings_data.rs	Define the logical default secret store and thread it through config-store loading.
crates/trusted-server-integration-tests/Cargo.toml	Make TOML parsing available to the integration config generator.
crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml	Replace integration fixture credentials with secret key names.
crates/trusted-server-integration-tests/fixtures/configs/viceroy-template.toml	Add the local runtime mapping and fictional secret-store entries used by Viceroy.
crates/trusted-server-integration-tests/src/bin/generate-viceroy-config.rs	Generate local secret-store data for references found in integration configuration.
crates/trusted-server-integration-tests/tests/common/config.rs	Build test envelopes from the typed application-config representation.
crates/trusted-server-integration-tests/tests/environments/axum.rs	Supply fictional referenced secrets to the Axum integration environment.
docs/guide/asset-routes.md	Update asset-route examples to use the resolved publisher secret model.
docs/guide/configuration.md	Explain reference syntax, conditional requirements, optional partner API access, compatibility behavior, redaction, and migration from feature-specific stores.
docs/guide/ec-setup-guide.md	Clarify that the demo requires a partner API token because it exercises inbound identify and batch-sync APIs, while other partners may omit it.
docs/guide/fastly.md	Document Fastly's logical trusted_server_secrets to physical ts_secrets mapping and provisioning requirements.
docs/guide/getting-started.md	Add local setup instructions for config blobs and referenced secret values.
docs/guide/integrations/datadome.md	Replace the old DataDome store selector with a typed key reference.
fastly.toml	Configure the local Fastly runtime mapping and a placeholder physical secret store.
trusted-server.example.toml	Replace plaintext credentials with key names, mark partner API tokens as optional for partners that do not use inbound APIs, and add Tinybird and DataDome reference examples.

Scope

This PR touches the core schema, each adapter startup path, integration fixtures, and operator documentation because secret references must behave the same on Fastly, Axum, Cloudflare, and Spin. The request-signing key collection, rotation stores, and Fastly management credentials remain outside this change because those stores are managed at runtime rather than loaded as static application configuration.

EdgeZero dependency

This PR depends on stackpop/edgezero#344, "Support optional typed secret paths and Fastly store mappings." That PR adds optional intermediate path handling and persists validated logical-to-physical store mappings during Fastly provisioning and staged deployment. Trusted Server pins its tested commit, 0d6ebf9b0250efa5f7031a93ec7b7f09f2c9bf34. All checks on the EdgeZero PR pass.

Closes

Closes #684

Test plan

• cargo test-fastly && cargo test-axum
• cargo clippy-fastly && cargo clippy-axum
• cargo fmt --all -- --check
• JS tests: cd crates/trusted-server-js/lib && npx vitest run, no JS source changed
• JS format: cd crates/trusted-server-js/lib && npm run format, no JS source changed
• Docs format: cd docs && npm run format
• WASM build: the deployment workflow built and staged the Fastly artifact
• Manual testing via fastly compute serve
• Other: cargo test-cloudflare, cargo test-spin, adapter parity tests, CLI tests, Cloudflare and Spin WASM checks, all adapter-specific Clippy targets, and git diff --check
• Staged Fastly deployment: run 32784895487 passed /health; a settings-load probe found no secret-resolution or application-state errors

Checklist

• Changes follow CLAUDE.md conventions
• No unwrap() in production code, use expect("should ...")
• Logging follows project conventions; no direct stdout or stderr logging was added
• New code has tests
• No secrets or credentials committed

[aram356]

@ChristianPavilonis to test it before merging into #1019

[prk-Jr]

Summary

Moves static app-config credentials from plaintext blob values to secret-store key
references resolved after envelope verification, and fixes the logical-to-physical
store mapping that broke the Fastly deployment. The core design is sound: integrity
verification genuinely precedes resolution, resolution is atomic (the blob is left
untouched on failure), the deploy/load validation split keeps value checks on the load
path where PartnerRegistry::from_config still fails closed, and the two end-to-end
payload tests cover both the all-credentials-resolve and inactive-feature-skip arms.

Four blocking items: resolution discards the one diagnostic that would explain a
mis-mapped store, the documented migration order opens a total outage window, the
Fastly Hooks::routes() path reads the store mapping from the wrong source, and the
EdgeZero dependency is pinned to an unmerged upstream commit.

3 of the inline comments below carry a one-click GitHub suggestion — use
Commit suggestion (or Add suggestion to batch) to apply them as commits on
the PR branch. The remaining comments describe the fix in prose because the change
spans multiple files, needs a new import, or adds code outside the diff. No
suggestion in this review was scratch-verified — local runs were skipped for this
pass, so please re-run the matching checks after applying.

Blocking

🔧 wrench

• Secret-store resolution throws away every adapter's diagnostic — see inline at crates/trusted-server-core/src/secret_resolution.rs:164
• Documented migration order opens a full outage window — see Cross-cutting below
• Hooks::routes() reads the wrong source for the store mapping — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1261

❓ question

• EdgeZero pinned to an unmerged upstream PR — see Cross-cutting below

Non-blocking

♻️ refactor / 🤔 thinking / ⛏ nitpick / 🌱 seedling

• Required S3 secret references still have serde defaults — see inline at crates/trusted-server-core/src/settings.rs:767
• Feature-enablement logic duplicated in three places — see inline at crates/trusted-server-core/src/config_payload.rs:63
• EchoSecretStore makes resolution untestable — see inline at crates/trusted-server-core/src/config_payload.rs:145
• expect() on the Tinybird token traps the Wasm guest — see inline at crates/trusted-server-adapter-fastly/src/tinybird.rs:57
• Deploy validation misses duplicate partner key names — see inline at crates/trusted-server-core/src/ec/registry.rs:74
• New docs bullets lost their markdown hard breaks — see inline at docs/guide/configuration.md:1620
• partners = [] is redundant and a footgun — see inline at trusted-server.example.toml:17
• Two overlapping ways to express leaf optionality — see inline at crates/trusted-server-core/src/secret_resolution.rs:64
• Spin's five declared secret variables read as a contract — see inline at crates/trusted-server-adapter-spin/spin.toml:28

Cross-cutting / body-level findings

• 🔧 Documented migration order opens a full outage window — docs/guide/configuration.md:60-72 gives the order: populate store, replace values with key names, ts config validate + ts config push, then "restart/redeploy instances as needed."

  Step 3 lands the reference-bearing blob while the old binary is still serving. On Fastly each request reads the config store fresh, so from that instant every request runs Ec::validate_passphrase — which requires at least 32 bytes on main today (MIN_PASSPHRASE_LENGTH = 32, crates/trusted-server-core/src/settings.rs) — against passphrase = "ec_passphrase" (13 bytes). That yields short_passphrase, config load fails, and the service returns its startup-error response for all traffic until the redeploy finishes.

  The reverse mismatch fails too: a new binary reading a plaintext blob resolves each plaintext secret as a key name. There is no safe intermediate state — the binary and the blob have to flip together, and the doc currently puts the break in the middle. Please correct the ordering and add an explicit warning that a mismatched binary/blob pair fails config load outright. The staged Fastly deployment cited in the PR description would not surface this, since no old binary is in play there.

• ❓ EdgeZero pinned to an unmerged upstream PR — Cargo.toml:57-62 moves all six edgezero crates from stable tag v0.0.4 to git rev 0d6ebf9b0250efa5f7031a93ec7b7f09f2c9bf34, a commit on the still-open stackpop/edgezero#344. Merging this puts main on a branch commit of an unmerged PR: if #344 is rebased or force-pushed before it merges, that commit can become unreachable and main stops building.

  You disclosed this in the PR description, and the issue comment suggests this lands in #1019 first, so this may already be handled. It still needs an explicit answer because it constrains main: hold this PR until #344 merges and re-pin to a tag, or is a rev pin on main acceptable here?

• 📝 CI coverage gap on the reviewed head — only Analyze (javascript-typescript) ran on 1315cdb1. The full gate suite (cargo fmt/test/clippy, all four adapters, cross-adapter parity, vitest, format-docs, integration and browser tests) last ran green on the merge commit 598f7100, three commits earlier. That leaves 070397f1 Resolve static credentials through typed config, b1e967e3, and 1315cdb1 without Rust, adapter, or lint coverage. Worth re-triggering the suite on the current head before merge, independent of the findings above.

• 👍 validation_error_summary is a real leak fix — crates/trusted-server-core/src/settings.rs:2387-2424 walks ValidationErrors emitting only path: code, never validator's params, which hold the offending value. The previous code formatted ValidationErrors wholesale into a config error message.

• 👍 Deleting S3_CREDENTIALS_CACHE removes a genuinely bad structure — crates/trusted-server-core/src/proxy.rs previously kept a process-global HashMap keyed on the plaintext secret access key, with unbounded growth and a poisoning-prone Mutex. Startup-resolved values are strictly better.

• 👍 IntegrationSettings's custom Debug closes the DataDome-key leak that the flattened JsonValue map would otherwise print.

• 👍 The two payload resolution tests are the right pair — resolves_all_static_credentials_from_the_mapped_default_store proves every path arm resolves through a mapped physical store, and inactive_optional_features_do_not_resolve_stale_secret_references proves disabled features do not demand stale references. Also good: dropping include_str!("trusted-server.example.toml") from the Spin and Cloudflare startup paths in favour of a hard error.

CI Status

• Analyze (javascript-typescript): PASS
• cargo fmt: not run on this head (PASS on 598f7100)
• cargo test: not run on this head (PASS on 598f7100)
• cargo test (axum native): not run on this head (PASS on 598f7100)
• cargo test (cross-adapter parity): not run on this head (PASS on 598f7100)
• cargo test (ts CLI, native): not run on this head (PASS on 598f7100)
• cargo check (cloudflare native + wasm32-unknown-unknown): not run on this head (PASS on 598f7100)
• cargo check/build/test (spin native + wasm32-wasip1): not run on this head (PASS on 598f7100)
• integration tests: not run on this head (PASS on 598f7100)
• integration tests (Fastly EC lifecycle): not run on this head (PASS on 598f7100)
• browser integration tests: not run on this head (PASS on 598f7100)
• prepare integration artifacts: not run on this head (PASS on 598f7100)
• vitest: not run on this head (PASS on 598f7100)
• format-typescript: not run on this head (PASS on 598f7100)
• format-docs: not run on this head (PASS on 598f7100)
• Analyze (rust): not run on this head (PASS on 598f7100)
• Analyze (actions): not run on this head (PASS on 598f7100)
• CodeQL: not run on this head (PASS on 598f7100)

No check reported a fail or cancel bucket. Branch protection reported no required checks for this PR.

[ChristianPavilonis]

Review follow-up for b47ced81a:

• The accepted diagnostics, DataDome pruning, Fastly mapping, EC validation, harness, example, and documentation changes are implemented. I replied to and resolved all 11 inline threads.
• I am not adding migration orchestration for the plaintext-to-reference transition. Trusted Server has not entered production with the old plaintext configuration model, so secret references will be the production baseline rather than a live migration. Deployment machinery here would be speculative.
• The EdgeZero revision remains an intentional dependency for RC validation. PR Support optional typed secret paths and Fastly store mappings stackpop/edgezero#344 is still the upstream dependency; Add native secret-store config resolution #1036 should use its accepted final revision before a mainline merge if project policy requires that.
• Local validation passed: all four adapter test suites, all six Clippy targets, Rust and docs formatting, git diff --check, focused regression tests, and both template-cache harness modes. GitHub checks are rerunning on the pushed head.

Re-requesting review from @prk-Jr.

[aram356]

Summary

Well-executed change: the resolution model (verify envelope, strip inactive references, resolve, validate runtime settings) is fail-closed, the push-time/runtime validation split is coherent across all four adapters, and the test coverage in config_payload.rs and secret_resolution.rs is thorough. Two blocking findings: a secret-exposure path in the resolution-failure error message, and the failed CodeQL check.

4 of the inline comments below carry a one-click GitHub suggestion — use Commit suggestion (or Add suggestion to batch) to apply them as commits on the PR branch. The remaining comments describe the fix in prose because the change touches multiple locations and can't be auto-applied.

Blocking

🔧 wrench

• Resolution-failure error can log a plaintext secret from a legacy blob — see inline at crates/trusted-server-core/src/secret_resolution.rs:169
• CodeQL check failed: 15 high rust/cleartext-logging alerts — see Cross-cutting below

Non-blocking

♻️ refactor / 🤔 thinking / ⛏ nitpick / 📝 note / 🌱 seedling

• .env.example ships the Fastly store mapping active, breaking the documented Axum flow — see inline at .env.example:11 (suggestion)
• Migration guide doesn't warn that previously legal short secrets now fail startup — see inline at docs/guide/configuration.md:70 (suggestion)
• Missing required leaf reports "must be a string" instead of "missing" — see inline at crates/trusted-server-core/src/secret_resolution.rs:151 (suggestion)
• server_side_key_secret_name holds the resolved key value at runtime — see inline at crates/trusted-server-core/src/integrations/datadome.rs:184 (suggestion)
• validate_config_for_deploy uses HashMap<_, ()> as a set — see inline at crates/trusted-server-core/src/ec/registry.rs:77
• Hooks::stores() duplicates edgezero.toml — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1332
• Cargo.lock rewrote prost's itertools edges — see Cross-cutting below

Cross-cutting / body-level findings

• 🔧 CodeQL check failed: 15 high rust/cleartext-logging alerts. Not required under branch protection, but a CI gate this repo treats as blocking. I inspected all 15: they are taint over-approximation — CodeQL now treats everything flowing out of resolve_secret_references / validate_tinybird_secret / validate_admin_handler_passwords as secret-tainted and flags logs of plainly non-secret fields (asset-route prefixes in settings.rs, DataDome registration flags in datadome.rs:979, consent clamping in consent_config.rs, header names in response_privacy.rs, etc.). No alert is a real value leak — the nearest real vector is the inline finding at secret_resolution.rs:169. The alerts still need triage: dismiss each in the code-scanning UI with a justification (or add a CodeQL model/sanitizer exclusion), otherwise this check stays red here and re-fires on every future PR touching these paths.
• ⛏ Cargo.lock moved prost's itertools dependency edges from 0.13.0 to 0.10.5. The edgezero pin update also rewrote prost-build/prost-derive's itertools edges down to the already-present 0.10.5 while 0.13.0 stays in the graph for other consumers — unintended churn from edge unification during the scoped update. Consider hand-restoring the 0.13.0 edges so the lock diff stays scoped to the edgezero bump.

CI Status

• browser integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• integration tests: PASS
• CodeQL: FAIL
• cargo test (ts CLI, native): PASS
• cargo test (cross-adapter parity): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• format-docs: PASS (required)
• cargo test: PASS (required)
• cargo test (axum native): PASS
• format-typescript: PASS (required)
• Analyze (rust): PASS
• Analyze (javascript-typescript): PASS
• Analyze (javascript-typescript): PASS
• cargo fmt: PASS (required)
• prepare integration artifacts: PASS
• vitest: PASS
• Analyze (actions): PASS

[aram356]

Summary

Second pass, reviewing head 76f6f13. The feedback commit addresses every finding from the previous review: the resolution-failure error now drops both the key name and the underlying platform error (with a regression test asserting a legacy plaintext value never reaches diagnostics), the new 32-byte minimums were removed in favor of pre-PR behavior (bypass-credential strength enforcement moved back to request time, with a request-level test), .env.example no longer ships the Fastly mapping active, and the stores() metadata is now pinned to edgezero.toml by a manifest-parsing test. What remains blocking is the open CodeQL alert set; one stale doc claim and the lockfile nit round out the list.

1 of the inline comments below carries a one-click GitHub suggestion — use Commit suggestion to apply it as a commit on the PR branch.

Blocking

🔧 wrench

• CodeQL: 15 high rust/cleartext-logging alerts still open — see Cross-cutting below

Non-blocking

⛏ nitpick

• Stale "must be at least 32 bytes" claim for proxy_secret — see inline at docs/guide/configuration.md:361 (suggestion)
• Cargo.lock prost itertools edges still rewritten — see Cross-cutting below

Cross-cutting / body-level findings

• 🔧 CodeQL: 15 high rust/cleartext-logging alerts still open. Carried over from the previous review round. The code fix in 76f6f13 does not clear them — all 15 are taint over-approximation (CodeQL treats everything flowing out of resolve_secret_references / validate_tinybird_secret / validate_admin_handler_passwords as secret-tainted and flags logs of plainly non-secret fields), and all 15 remain open on this PR, so the CodeQL check will fail again once analysis reruns on this head. They need triage: dismiss each in the code-scanning UI with a justification, or add a CodeQL suppression/model exclusion — otherwise this check stays red here and re-fires on every future PR touching these paths.
• ⛏ Cargo.lock still carries the rewritten prost itertools edges (0.13.0 → 0.10.5 while 0.13.0 stays in the graph for other consumers) — unaddressed nit from the previous review; the earlier inline thread on this stays open, so no new inline comment here. Hand-restoring the 0.13.0 edges keeps the lock diff scoped to the edgezero bump.

CI Status

GitHub checks have not yet run for head 76f6f13 — only one check has reported; everything else is pending/not started. Local verification was run in the reviewer worktree at this head instead: cargo fmt --all -- --check, cargo clippy-fastly, targeted cargo test-fastly for the modules this head touches (10 secret_resolution + 15 config_payload + 67 datadome + 43 registry + the fastly manifest-metadata test), and prettier for the changed docs — all pass.

• Analyze (javascript-typescript): PASS
• CodeQL: not run on this head (15 alerts from the prior analysis remain open)
• browser integration tests: not run
• integration tests (Fastly EC lifecycle): not run
• integration tests: not run
• cargo test (ts CLI, native): not run
• cargo test (cross-adapter parity): not run
• cargo check/build/test (spin native + wasm32-wasip1): not run
• cargo check (cloudflare native + wasm32-unknown-unknown): not run
• format-docs: not run (required; passes locally)
• cargo test: not run (required; touched modules pass locally)
• cargo test (axum native): not run
• format-typescript: not run (required)
• Analyze (rust): not run
• cargo fmt: not run (required; passes locally)
• prepare integration artifacts: not run
• vitest: not run
• Analyze (actions): not run

[aram356]

⛏ nitpick — This still promises a hard requirement that 76f6f13 removed: MIN_PROXY_SECRET_LENGTH and validate_proxy_secret_strength are gone, and config_payload.rs now has a test asserting a short resolved proxy secret loads successfully. proxy-signing.md was already softened to "recommended" — this section should match, or operators will expect a startup failure that no longer happens.

Suggested change

	The referenced value is resolved from `trusted_server_secrets` at startup. It
	must be at least 32 bytes, so generate it with a cryptographically secure random
	source. Keep that value confidential, rotate it only intentionally, and never
	put it in the TOML file or pushed app-config blob.
	The referenced value is resolved from `trusted_server_secrets` at startup.
	Generate it with a cryptographically secure random source; at least 32 random
	bytes are recommended. Keep that value confidential, rotate it only
	intentionally, and never put it in the TOML file or pushed app-config blob.