E2E Test Runner

HyperShift E2E Test Runner

This skill enables autonomous iteration on e2e tests - running tests, analyzing failures, making fixes, and re-running until tests pass.

When to Use This Skill

This skill automatically applies when:

• Implementing a feature that needs e2e test validation
• Fixing a failing e2e test
• Working on a task where the user wants you to iterate until tests pass
• Debugging test failures in the test/e2e/ directory
• The user mentions running e2e tests or validating changes against a live cluster

Prerequisites

Source the environment file before using this skill:

source dev/claude-env.sh

Environment Configuration

Environment variables from dev/claude-env.sh:

| Variable | Description |

|----------|-------------|

| E2E_PLATFORM | Test platform (AWS, Azure, etc.) |

| AWS_CREDENTIALS | Path to AWS credentials file |

| OIDC_BUCKET | S3 bucket for OIDC |

| BASE_DOMAIN | Base DNS domain |

| PULL_SECRET | Path to pull secret file |

| AWS_REGION | AWS region |

| E2E_ARTIFACT_DIR | Directory for test artifacts |

| MGMT_KUBECONFIG | Path to management cluster kubeconfig |

| CPO_IMAGE_REPO | Custom CPO image repository |

| RUNTIME | Container runtime (podman/docker) |

Running E2E Tests

Step 1: Check if Test Binary Needs Rebuilding

CRITICAL: Before running any e2e test, you MUST check if the test binary needs rebuilding:

# Check if binary exists
if [ ! -f ./bin/test-e2e ]; then
  echo "Test binary missing, building..."
  make e2e
fi

# Check if any test files are newer than the binary
NEWEST_TEST=$(find test/e2e -name "*.go" -newer ./bin/test-e2e 2>/dev/null | head -1)
if [ -n "$NEWEST_TEST" ]; then
  echo "Test files changed (e.g., $NEWEST_TEST), rebuilding..."
  make e2e
fi

Step 2: Run the Test

Build and execute the test command:

KUBECONFIG=$MGMT_KUBECONFIG \
./bin/test-e2e -test.v -test.timeout 2h \
  -test.run "TEST_PATTERN" \
  -test.v \
  --e2e.platform $E2E_PLATFORM \
  --e2e.aws-credentials-file $AWS_CREDENTIALS \
  --e2e.aws-oidc-s3-bucket-name $OIDC_BUCKET \
  --e2e.base-domain $BASE_DOMAIN \
  --e2e.pull-secret-file $PULL_SECRET \
  --e2e.aws-region $AWS_REGION \
  --e2e.artifact-dir $E2E_ARTIFACT_DIR

Step 3: Add Custom CPO Image (When Testing Control Plane Changes)

If you've made changes to control-plane-operator code and built a custom image, add:

-e2e.control-plane-operator-image $CPO_IMAGE_REPO:TAG

Iteration Loop

When working autonomously on a task that requires e2e validation:

1. Initial Test Run

Run the test to establish baseline:

KUBECONFIG=$MGMT_KUBECONFIG ./bin/test-e2e -test.v -test.run "TestName" [flags...]

2. On Failure - Analyze

• Read the test output carefully
• Check artifacts in $E2E_ARTIFACT_DIR/ directory for:
• Pod logs
• Events
• Resource states
• Identify the root cause

3. Make Fixes

• Edit the relevant code (test code, operator code, etc.)
• If you modified test/e2e/*.go files, the binary will be rebuilt automatically on next run

4. Rebuild Images (If Needed)

If you modified control-plane-operator code:

Use the build-cpo-image skill to build and push a new image.

$RUNTIME build -f Dockerfile.control-plane --platform linux/amd64 -t $CPO_IMAGE_REPO:NEW_TAG .
$RUNTIME push $CPO_IMAGE_REPO:NEW_TAG

5. Re-run Test

Run the test again with updated code/images. Repeat until passing.

Common Test Patterns

| Test Pattern | Description |

|--------------|-------------|

| TestNodePool | All NodePool tests |

| TestNodePool/HostedCluster0/Main/TestSpotTerminationHandler | Specific spot test |

| TestNodePool.*Karpenter | All Karpenter-related tests |

| TestCreateCluster | Cluster creation tests |

| TestUpgrade | Upgrade tests |

Analyzing Test Failures

Check Test Output

The test output includes:

• Test name and status
• Assertion failures with expected vs actual
• Timeout information
• Resource creation/deletion logs

Check Artifact Directory

After a test failure, examine:

ls -la $E2E_ARTIFACT_DIR/
# Contains: cluster manifests, pod logs, events, resource dumps

Common Failure Patterns

| Pattern | Likely Cause |

|---------|--------------|

| context deadline exceeded | Resource didn't reach expected state in time |

| not found | Resource wasn't created or was deleted prematurely |

| connection refused | Service not ready or network issue |

| forbidden | RBAC or permission issue |

Building Test Binary

When test code changes, rebuild:

make e2e

This compiles ./bin/test-e2e with all tests from test/e2e/.

Notes

• Tests typically take 10-30+ minutes depending on complexity
• Some tests create real AWS resources (costs money, needs cleanup on failure)
• Use -test.timeout to set appropriate timeouts (default: 2h)
• The artifact directory is overwritten on each run
• For long tests, consider running in background and checking periodically