Skip to content

e2e-testing.md

Latest commit

 

History

History
124 lines (100 loc) · 7.03 KB

e2e-testing.md

File metadata and controls

124 lines (100 loc) · 7.03 KB
Raw

E2E Testing Harness

End-to-end tests deploy a full GKG stack on a shared GKE cluster and run Robot Framework tests against it.

Architecture

Each run gets isolated namespaces keyed by commit SHA (e2e-{sha}-*). The stack includes GitLab (with ClickHouse migrations), NATS, ClickHouse, Siphon CDC, and all four GKG modes (webserver, indexer, dispatcher, health-check).

┌──────────┐    CDC     ┌──────┐   stream   ┌────────────┐
│  GitLab  │──────────▶│Siphon│──────────▶│ ClickHouse │
│  (+ PG)  │           │      │   (NATS)   │ datalake   │
└──────────┘           └──────┘           └─────┬──────┘
     ▲                                          │
     │ gRPC (TLS)                          read │
     │                                          ▼
┌──────────┐                              ┌──────────┐
│  Robot   │─── Orbit query API ────────▶│   GKG    │
│  tests   │                              │ webserver│
└──────────┘                              └──────────┘

Cluster

  • GKE: gke_gl-knowledgegraph-prj-f2eec59d_us-central1-a_e2e-harness
  • Harness config: gitlab-org/orbit/orbit-e2e-harness: cluster bootstrap (cert-manager, GitLab Agent)
  • CI access: GitLab Agent e2e-harness-agent

Running

# Local
e2e/scripts/setup.sh          # deploy full stack
e2e/scripts/test.sh           # run Robot Framework tests
e2e/scripts/teardown.sh -y    # cleanup

# Specific SHA
E2E_SHA=abc1234 e2e/scripts/setup.sh
E2E_SHA=abc1234 e2e/scripts/test.sh

In CI the e2e job runs automatically on main and manually on MRs.

Test suites

Suite What it tests
01_setup_and_smoke.robot Bootstrap e2e-bot, enable orbit_gql_queries, provision the shared namespace, smoke-test the pipeline
02_indexing.robot Create projects, issues, notes, epics; assert SDLC nodes and edges land in Orbit
03_code_indexing.robot Push fixture repos; assert File/Definition/IMPORTS/DEFINES via Orbit
04_code_backfill.robot Enable KG on a populated namespace and verify backfill dispatches code indexing
05_role_scoped_authz.robot Issue #347 — aggregation queries enforce per-entity authz on the target node. Seeds a victim user with Reporter/Security Manager/Developer/Maintainer/nested-subgroup memberships and replays the original oracle matrix per role. Requires Ultimate (activated by setup) and a GitLab image past gitlab-org/gitlab@7e57f842dada (publishes role-tagged traversal IDs).
06_incremental_update.robot Rails-side note delete propagates; Orbit stops returning the tombstoned node
07_namespace_lifecycle.robot Disable retains indexed data (30-day grace); re-enable resumes indexing
08_private_redaction.robot Private project/issue redacted from a non-member, visible to admin
09_api_surface.robot Read-only Orbit endpoints: schema, CALL db.schema(), schema/format, graph_status, tools, commands
10_query_shapes.robot neighbors, path_finding, and llm (GOON) response format
11_security_graph.robot Vulnerability node plus IN_PROJECT/AUTHORED/OCCURRENCE_OF edges
12_membership_graph.robot MEMBER_OF (User→Group) and CREATOR (User→Project) edges
13_cross_namespace_traversal.robot Scoped-query traversal-path pruning must not drop cross-namespace related entities

Query language

Every suite sends GQL text to POST /api/v4/orbit/query. Suite 01 turns on the orbit_gql_queries feature flag for the instance, so Rails selects the GQL frontend for every user. With the flag on, Orbit rejects JSON DSL queries. See Orbit query frontend.

Parallel execution

The robot-runner job executes suites with pabot. Suite 01_setup_and_smoke runs alone first. It bootstraps credentials, provisions the shared namespace, and proves the pipeline reached steady state. Then every other suite runs in a parallel worker pool.

  • e2e/tests/ordering.txt defines the barrier: suites listed before #WAIT run first; everything else is auto-discovered. A new NN_name.robot file needs no registration. It joins the parallel pool automatically.
  • Suite 01 publishes the shared namespace through PabotLib parallel keys. Downstream suites adopt it via the Attach To Shared Fixture suite setup (gitlab.resource). That setup also mints a per-suite admin bot user. The Rails orbit_query rate limit (60 req/min) is scoped per user, so suites must not poll through one shared PAT. A new suite that needs credentials or the shared namespace must declare that setup (copy the header of any existing suite).
  • Suites must not depend on state created by other downstream suites, and instance-global mutations (feature flags, license) belong in 01 before the barrier.
  • Plain robot runs still work for local debugging: PabotLib degrades to an in-process value store, so robot tests/ executes the suites sequentially with identical semantics.
  • In CI the runner pod uses the prebaked e2e-robot image. The e2e-robot-image job builds it from e2e/Dockerfile.robot whenever that file changes, tagged by its content hash. Local runs default to python:3.14-slim and install Robot Framework at pod startup.

Setup phases

Phase Step What it does
1 orphan cleanup + secrets Clean cluster-scoped leftovers, generate per-run secrets, extract the cert-manager root CA
2 sync-cdc-tables.sh Regenerate Siphon CDC config from the siphon-ssot-tables package registry artifact for the pinned gitlab.ref (publish-siphon-tables.sh publishes it when the ref is bumped)
3 background pollers bootstrap-instance.sh (license + root PAT, gated on the migrations Job) and patch-ch-dicts.sh (DIRECT-layout dictionaries) launch early and overlap the deploy
4 helmfile sync Deploy every release; GitLab gates pg-cdc-setup and siphon, while gkg needs only NATS and ClickHouse and overlaps the GitLab boot
5 patch-ch-siphon-watermark.sh Add the watermark column (after sync — it enumerates tables the siphon consumer creates), then join the background pollers

On failure, CI dumps diagnostics in after_script before teardown; green runs skip the dump.

Data-driven CDC tables

e2e/config/siphon-layout.yaml is the single source of truth for CDC table definitions. It drives the PostgreSQL publication table list and the Siphon Helm values (table mapping, streams, dedup config). Add a table once there and all consumers update automatically.

Key files

  • e2e/helmfile.yaml.gotmpl: all Helm releases
  • e2e/values/: per-component Helm values (.gotmpl for templated ones)
  • e2e/charts/: local charts (ClickHouse, robot-runner)