Skip to content

Wire transaction events into devnet#3633

Open
niran wants to merge 2 commits into
mainfrom
niran/txobs-devnet-vector
Open

Wire transaction events into devnet#3633
niran wants to merge 2 commits into
mainfrom
niran/txobs-devnet-vector

Conversation

@niran

@niran niran commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds devnet Vector configuration to tail transaction event JSONL files and ship NDJSON to audit-archiver.
  • Drops parsed devnet transaction events in Vector when data contains unsafe transaction/body/header key names, matching the production sidecar behavior.
  • Enables transaction event journal paths for devnet services and adds a proxyd transaction-events image/config for local verification.
  • Adds a smoke script that exercises devnet transactions and checks audit transaction event ingestion.
  • Documents component_discarded_events_total as the local Vector metric to inspect for malformed JSONL parse drops and unsafe-key validation drops.

Verification

  • docker run --rm -v /Users/niran/workspace/base/.worktrees/txobs-devnet-vector/etc/docker/transaction-events-vector.yaml:/etc/vector/vector.yaml:ro -v /private/tmp/txobs-devnet-vector-data:/vector-data timberio/vector:0.47.0-debian validate /etc/vector/vector.yaml

Stack Context

This PR is based on #3628 and contains only devnet/Vector wiring for local verification. It is intentionally reviewed separately from service producer code.

type=routine
risk=low
impact=sev5

@niran niran force-pushed the niran/txobs-devnet-vector branch from be7562f to 8d22450 Compare June 18, 2026 17:11
@niran niran force-pushed the niran/txobs-builder-events branch from 0887a65 to 1e55997 Compare June 18, 2026 17:11
@niran niran changed the base branch from niran/txobs-builder-events to niran/txobs-event-foundation June 18, 2026 17:11
COPY --from=base-routing proxyd/ ./proxyd/
RUN cd proxyd && make proxyd

FROM 652969937640.dkr.ecr.us-east-1.amazonaws.com/containers/golang:v1-24-ub22

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runtime stage uses the full Go SDK image (golang:v1-24-ub22) rather than a minimal base. The original Dockerfile.proxyd uses alpine:3.20 for the runtime, which is orders of magnitude smaller. Since the runtime only needs the proxyd binary (and curl for healthchecks), consider using a slim base image to reduce image size and attack surface.

Also worth noting: the healthcheck in docker-compose.ingress.yml runs curl, but this Dockerfile doesn't explicitly install it. It may work because the Ubuntu-based Go image likely ships curl, but this is an implicit dependency — if the base image changes, the healthcheck will silently break.

base_meteredPriorityFeePerGas = "client"

[transaction_events]
enabled = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TOML config sets enabled = false, but the docker-compose.ingress.yml sets TRANSACTION_EVENTS_ENABLED=true as an environment variable. If proxyd uses env-var overrides for its TOML config, this works as intended (TOML default = off, env override = on). But if proxyd reads the TOML file without env-var override support, transaction events won't be enabled despite the env var.

Could you confirm that proxyd supports environment variable overrides for the [transaction_events] TOML section? If it does, this pattern is fine. If not, this should be enabled = true (or the env var approach is dead code).

type: http
inputs:
- parse_transaction_events
uri: http://audit-archiver:9100/v1/transaction-events/batch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The audit-archiver port is hardcoded to 9100 here. While Vector YAML doesn't support Docker Compose variable interpolation, this creates a silent coupling with AUDIT_RPC_PORT=9100 in devnet-env. If someone changes AUDIT_RPC_PORT, they'd also need to find and update this file — consider adding a comment noting this dependency.

@niran niran force-pushed the niran/txobs-devnet-vector branch 2 times, most recently from f36f90d to 0d01740 Compare June 18, 2026 17:33
@niran niran force-pushed the niran/txobs-event-foundation branch 4 times, most recently from 9c11260 to 5f2ba67 Compare June 22, 2026 16:59
Base automatically changed from niran/txobs-event-foundation to main June 22, 2026 18:28
@cb-heimdall

Copy link
Copy Markdown
Collaborator

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

@niran niran requested a review from wlawt June 22, 2026 20:02
@niran niran force-pushed the niran/txobs-devnet-vector branch from 7fcff07 to 1d27433 Compare June 23, 2026 18:49
- ../../.devnet/transaction-events:/var/log/transaction-events
environment:
- BUILDER_TRANSACTION_EVENTS_ENABLED=true
- TRANSACTION_EVENTS_PATH=/var/log/transaction-events/base-builder/events.jsonl

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The builder env vars use the BUILDER_ prefix for TRANSACTION_EVENTS_ENABLED and TRANSACTION_EVENTS_NETWORK, but TRANSACTION_EVENTS_PATH has no prefix. Compare with base-client below, where all three use the BASE_ prefix consistently (BASE_TRANSACTION_EVENTS_ENABLED, BASE_TRANSACTION_EVENTS_PATH, BASE_TRANSACTION_EVENTS_NETWORK).

Is the builder intentionally using a bare TRANSACTION_EVENTS_PATH, or should this be BUILDER_TRANSACTION_EVENTS_PATH?

@github-actions

Copy link
Copy Markdown
Contributor

Review Summary

This PR wires transaction event observability into the devnet: Vector config for JSONL tailing, Postgres + audit-archiver for persistence, proxyd transaction-events Dockerfile, and a smoke test script. The changes are mostly Docker Compose / infrastructure wiring with no Rust code changes (beyond a single CLI flag addition in docker-compose.yml).

Findings

Inconsistent env var prefix on base-builder (docker-compose.ingress.yml:13): BUILDER_TRANSACTION_EVENTS_ENABLED and BUILDER_TRANSACTION_EVENTS_NETWORK use the BUILDER_ prefix, but TRANSACTION_EVENTS_PATH does not. The base-client section uses the BASE_ prefix consistently for all three vars. This may be intentional if the builder binary expects a bare TRANSACTION_EVENTS_PATH, but it's worth confirming — if it's a bug, the builder won't find the journal path.

Notes (not blocking)

  • The --enable-transaction-event-journal flag added to base-client in the base docker-compose.yml is a no-op for non-ingress devnet runs since no path is configured. The writer gracefully degrades (required: false default), so this is safe but will produce a warning log on each non-ingress devnet startup.
  • The Vector unsafe_data_key_regex is a superset of the Rust-side FORBIDDEN list (adds patterns like bearer_tokens, private_keys, tx_body, etc.). This is fine as defense-in-depth — the Vector layer is a second filter.
  • The docker-bake.hcl refactoring into DEVNET_TARGETS + INGRESS_EXTRA_TARGETS variables with concat() is a clean improvement that avoids listing targets in two places.

@github-actions

Copy link
Copy Markdown
Contributor

✅ base-std fork tests: all 616 passed

base/base is fully in sync with the base-std spec.

Dependency Ref Commit
base-std main 4658f1b7
base-anvil 0092692587d8d064dd2c6923ce26a682c58f3694 00926925

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants