E2E v2 Test Flow

This document describes the end-to-end flow of the HyperShift v2 e2e test framework, from CI job trigger through test execution and teardown. It covers process boundaries, inter-process communication, and the sequencing of mutually exclusive tests.

Contents

• Ginkgo Decorators, Hooks, and Labels for Test Isolation
  • Decorators
  • Hooks
  • Labels
  • How These Layers Compose
• High-Level Flow
• Inside a test-e2e-v2 Process (Ginkgo Lifecycle)
• Process Boundary Summary
• Sequencing of Mutually Exclusive Tests
• Inter-Process Communication

Ginkgo Decorators, Hooks, and Labels for Test Isolation

The v2 framework uses Ginkgo features at two levels to keep tests from interfering with each other: the run-tests orchestrator isolates test groups into separate OS processes targeting different clusters, and within each process, Ginkgo decorators and hooks manage execution order, state mutation, cleanup, and reporting semantics.

Decorators

Ordered is the primary tool for inter-test dependencies within a single feature (e.g., backup must complete before restore can start). Serial adds the guarantee that no other spec in the process runs at the same time, which matters for tests that mutate cluster-wide resources like HostedCluster configuration or etcd state. In practice, since run-tests does not pass --procs to Ginkgo, all specs within a process already run sequentially — but Serial makes the constraint explicit and future-proof.

Hooks

The BeforeAll/AfterAll pair is critical for lifecycle tests that share expensive preconditions across multiple ordered specs (e.g., backup-restore creates a backup once, then multiple specs verify different aspects of the restore). Without Ordered, BeforeAll/AfterAll cannot be used — Ginkgo enforces this at the framework level.

Labels

How These Layers Compose

run-tests orchestrator
├── Process 1 (public cluster): --ginkgo.label-filter="self-managed-azure-public || nodepool-lifecycle || ..."
│   ├── Describe "NodePool Lifecycle" [Ordered] ← specs run in order, share BeforeAll setup
│   │   ├── BeforeAll: create test nodepool
│   │   ├── It "should scale up" ← mutation test
│   │   ├── It "should scale down"
│   │   └── AfterAll: delete test nodepool
│   ├── Describe "Control Plane Workloads" ← read-only, no Ordered needed
│   │   ├── It "should have resource requests" ← stateless assertion
│   │   └── Context "Custom labels" [Informing] ← failure → skip, non-blocking
│   └── ...
├── Process 2 (private cluster): --ginkgo.label-filter="self-managed-azure-private || ..."
│   └── ...
└── Sequential group (upgrade cluster):
    ├── Process 6a: --ginkgo.label-filter="control-plane-upgrade" ← must finish before 6b
    │   └── Describe "Control Plane Upgrade" ← triggers version rollout
    └── Process 6b: --ginkgo.label-filter="etcd-chaos" ← only runs if 6a passed
        └── Describe "Etcd Chaos" [Ordered] ← specs run in order, BeforeAll snapshots etcd

Cluster-level isolation (different processes target different clusters) prevents inter-group interference. Within a process, Ordered/Serial prevent inter-spec interference for mutation-heavy features. DeferCleanup ensures each spec restores what it touched. Informing decouples experimental coverage from gate status. The lifecycle label separates mutation tests from read-only compliance runs at the CI job level.

High-Level Flow

The diagram below shows the general v2 e2e flow. The framework is platform-agnostic — each platform implements the PlatformConfig interface — but Azure is currently the only implementation and serves as the reference. The concrete examples here follow the e2e-azure-v2-self-managed CI job and its workflow. ci-operator builds the hypershift-tests image (via Dockerfile.e2e, which invokes several Makefile targets), then chains together cluster creation, test execution, and teardown steps.

Inside a test-e2e-v2 Process (Ginkgo Lifecycle)

Each test-e2e-v2 invocation is a single OS process running the Ginkgo v2 test framework. The process is a compiled Go test binary (go test -c) with the e2ev2 build tag.

Process Boundary Summary

Sequencing of Mutually Exclusive Tests

Mutual exclusion between test groups is achieved through cluster isolation and sequential groups, not through in-process locking:

Key mechanisms:

1. Cluster-per-group isolation: Each parallel test group targets a different HostedCluster. Tests within a group share one cluster but different groups never touch the same cluster. This eliminates inter-group interference without locks.

2. Label-based partitioning: Ginkgo's --ginkgo.label-filter ensures each test-e2e-v2 process only runs specs matching its assigned labels. The label sets are non-overlapping across groups, so the same spec never runs in two processes.

3. Sequential groups for ordered dependencies: The upgrade-and-chaos sequential group runs upgrade first, then etcd-chaos on the same cluster. The run-tests orchestrator enforces ordering by running steps sequentially within a single goroutine. If upgrade fails, etcd-chaos is skipped (the goroutine returns early).

4. No in-process mutex: Because each test-e2e-v2 process targets exactly one cluster and runs non-overlapping label sets, there is no need for mutexes or other synchronization between test specs. Ginkgo runs specs within a single process serially by default (no --procs flag is passed).

Inter-Process Communication

• SHARED_DIR: Filesystem directory shared across all CI steps within a job. create-guests writes cluster names and platform-specific config; run-tests reads them. This is the primary IPC mechanism between CI steps.
• Environment variables: run-tests passes cluster identity to each test-e2e-v2 subprocess via E2E_HOSTED_CLUSTER_NAME and E2E_HOSTED_CLUSTER_NAMESPACE env vars.
• PROW_JOB_ID + SHA256: destroy-guests does not read SHARED_DIR cluster names. Instead, it re-derives cluster names deterministically from PROW_JOB_ID using the same DeriveClusterName() function as create-guests. This makes teardown idempotent and independent of whether creation succeeded.
• KUBECONFIG: All processes authenticate to the management cluster via the kubeconfig file at ${SHARED_DIR}/management_cluster_kubeconfig, set up by the nested management cluster provisioning step.
• Exit codes: run-tests collects exit codes from all test-e2e-v2 subprocesses and exits non-zero if any group failed.
• JUnit XML: Each test-e2e-v2 process writes a separate JUnit report to ARTIFACT_DIR. ci-operator collects these for Sippy/Prow reporting.