refactor: redesign E2E to use custom Docker image instead of PVC/kubectl

Replace the PVC + kubectl-patch approach for E2E plugin deployment with
a custom Docker image that has the plugin pre-installed. This eliminates
all policy-violating operations:

- No PVCs in kube-system
- No kubectl exec/cp to Headlamp pods
- No deployment patching via kubectl
- No temporary pods or ConfigMap-based file transfers

The new approach builds a Headlamp image with the plugin baked in
(Dockerfile.e2e), deploys it as a dedicated instance in the headlamp-e2e
namespace via Helm, and tears it down after tests complete.

RBAC is scoped to the headlamp-e2e namespace instead of kube-system.

Note: .github/workflows/e2e.yaml still needs updating to use the new
scripts — that change is delegated to Hugh (CI/CD owner).

Closes: privilegedescalation/headlamp-polaris-plugin#72

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Gandalf the Greybeard
2026-03-20 00:33:09 +00:00
parent 4296eb97fb
commit 6189f2b983
9 changed files with 219 additions and 221 deletions
+1
View File
@@ -6,5 +6,6 @@ e2e/.auth/
test-results/
.playwright-mcp/
.env
.env.e2e
.env.local
.eslintcache
+16
View File
@@ -0,0 +1,16 @@
# Dockerfile.e2e
#
# Builds a Headlamp image with the polaris plugin pre-installed.
# Used by E2E tests — not for production distribution (use ArtifactHub).
#
# Usage:
# npm run build
# docker build -f Dockerfile.e2e -t ghcr.io/privilegedescalation/headlamp-polaris-e2e:sha-abc123 .
#
# The plugin dist/ must be built before running docker build.
ARG HEADLAMP_VERSION=latest
FROM ghcr.io/headlamp-k8s/headlamp:${HEADLAMP_VERSION}
COPY dist/ /headlamp/plugins/headlamp-polaris/
COPY package.json /headlamp/plugins/headlamp-polaris/
+20 -32
View File
@@ -1,57 +1,45 @@
---
# RBAC for the GitHub Actions CI runner to perform E2E test setup.
# RBAC for the GitHub Actions CI runner to manage the E2E Headlamp instance.
# CI-only test fixture — NOT for production use.
#
# Grants the ARC runner service account namespace-scoped permissions in
# kube-system to patch the Headlamp deployment (add shared volume mount),
# manage PVCs, run temporary pods, and restart deployments.
# Grants the ARC runner service account permissions in the headlamp-e2e
# namespace to deploy and tear down a dedicated Headlamp instance via Helm.
#
# No cluster-scoped permissions needed — the E2E workflow uses kubectl patch
# instead of helm upgrade, avoiding the need to read ClusterRole/ClusterRoleBinding.
# No kube-system access needed — E2E tests use a separate namespace.
# No PVC management — plugin is baked into the Docker image.
#
# Apply with: kubectl apply -f deployment/e2e-ci-runner-rbac.yaml
# Prerequisites:
# kubectl create namespace headlamp-e2e
# kubectl apply -f deployment/e2e-ci-runner-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: e2e-ci-runner
namespace: kube-system
namespace: headlamp-e2e
rules:
# Helm needs to manage these resources for the Headlamp chart
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "create", "update", "patch", "delete", "watch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
resources: ["services", "serviceaccounts", "configmaps", "secrets"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "create", "delete", "watch"]
verbs: ["get", "list", "watch"]
# Token creation for E2E test auth
- apiGroups: [""]
resources: ["pods/attach"]
verbs: ["create", "get"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "patch", "watch"]
- apiGroups: ["apps"]
resources: ["deployments/scale"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get", "list"]
resources: ["serviceaccounts/token"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: e2e-ci-runner-binding
namespace: kube-system
namespace: headlamp-e2e
subjects:
- kind: ServiceAccount
name: local-ubuntu-latest-gha-rs-no-permission
name: runners-privilegedescalation-gha-rs-no-permission
namespace: arc-runners
roleRef:
kind: Role
+17 -16
View File
@@ -1,22 +1,23 @@
---
# Headlamp Helm values for E2E testing with shared volume plugin deployment.
# Headlamp Helm values for E2E testing.
#
# The CI runner and Headlamp pod share a PVC so that the runner can copy
# built plugin artifacts directly into Headlamp's plugins directory.
# This is a CI-only mechanism — production plugin distribution uses ArtifactHub.
# Uses a custom Docker image (built from Dockerfile.e2e) with the plugin
# pre-installed. No PVCs, no volume mounts, no deployment patching.
#
# The E2E workflow builds the image, pushes to ghcr.io, and deploys this
# Helm release in the headlamp-e2e namespace.
#
# Usage:
# helm install headlamp-e2e headlamp/headlamp \
# -n headlamp-e2e --create-namespace \
# -f deployment/headlamp-e2e-values.yaml \
# --set image.registry=ghcr.io \
# --set image.repository=privilegedescalation/headlamp-polaris-e2e \
# --set image.tag=<sha>
# Point Headlamp at the shared plugins mount
config:
pluginsDir: /headlamp/plugins
watchPlugins: false
# PVC-backed volume shared with the CI runner
volumes:
- name: plugins
persistentVolumeClaim:
claimName: headlamp-plugins
# Mount into the Headlamp container
volumeMounts:
- name: plugins
mountPath: /headlamp/plugins
readOnly: true
service:
type: ClusterIP
-14
View File
@@ -1,14 +0,0 @@
---
# PVC for sharing built plugin artifacts between the CI runner and Headlamp.
# Used only in E2E test environments — not for production.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: headlamp-plugins
namespace: kube-system
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 128Mi
+33 -24
View File
@@ -4,7 +4,17 @@ Playwright-based smoke tests that validate the Polaris plugin against a live Hea
## CI
E2E tests run automatically in GitHub Actions on pushes to `main` and pull requests. The workflow (`.github/workflows/e2e.yaml`) uses either Authentik OIDC or token-based authentication via repository secrets.
E2E tests run automatically in GitHub Actions on pushes to `main` and pull requests. The workflow (`.github/workflows/e2e.yaml`):
1. Builds the plugin
2. Builds a custom Headlamp Docker image with the plugin pre-installed (`Dockerfile.e2e`)
3. Pushes the image to `ghcr.io/privilegedescalation/headlamp-polaris-e2e:<sha>`
4. Deploys a dedicated Headlamp instance in the `headlamp-e2e` namespace via Helm
5. Generates a ServiceAccount token for test auth
6. Runs Playwright tests against the E2E instance
7. Tears down the E2E instance
This approach avoids PVCs, kubectl exec/cp, and deployment patching. The plugin is part of the container image.
### Required GitHub Secrets
@@ -12,12 +22,10 @@ Configure these in GitHub repository settings (Settings → Secrets and variable
| Secret | Required | Description |
| -------------------- | -------- | -------------------------------------------------------------- |
| `HEADLAMP_URL` | Optional | Headlamp instance URL (defaults to `https://headlamp.animaniacs.farh.net`) |
| `AUTHENTIK_USERNAME` | OIDC | Authentik email or username for a CI user with Headlamp access |
| `AUTHENTIK_PASSWORD` | OIDC | Password for that user |
| `HEADLAMP_TOKEN` | Token | Kubernetes service account token (alternative to OIDC) |
Set either `AUTHENTIK_USERNAME` + `AUTHENTIK_PASSWORD` **or** `HEADLAMP_TOKEN`. OIDC takes priority if both are set.
Token-based auth is auto-generated by the deploy script. OIDC secrets are only needed if testing against the shared Headlamp instance.
## Running Locally
@@ -47,12 +55,12 @@ HEADLAMP_URL=http://localhost:4466 npm run e2e:headed
| Variable | Required | Default | Description |
| -------------------- | -------- | -------------------------------------- | --------------------------------------- |
| `HEADLAMP_URL` | No | `https://headlamp.animaniacs.farh.net` | Base URL of the Headlamp instance |
| `AUTHENTIK_USERNAME` | OIDC | — | Authentik email/username |
| `AUTHENTIK_PASSWORD` | OIDC | — | Authentik password |
| `HEADLAMP_TOKEN` | Token | — | Kubernetes bearer token (fallback auth) |
| `HEADLAMP_URL` | No | `https://headlamp.animaniacs.farh.net` | Base URL of the Headlamp instance |
| `AUTHENTIK_USERNAME` | OIDC | — | Authentik email/username |
| `AUTHENTIK_PASSWORD` | OIDC | — | Authentik password |
| `HEADLAMP_TOKEN` | Token | — | Kubernetes bearer token (auto-generated in CI) |
Set either `AUTHENTIK_USERNAME` + `AUTHENTIK_PASSWORD` or `HEADLAMP_TOKEN`. OIDC takes priority if both are set.
In CI, `HEADLAMP_URL` and `HEADLAMP_TOKEN` are set automatically by the deploy script. For local runs, set either OIDC credentials or a token manually.
## What the Tests Validate
@@ -249,25 +257,26 @@ test('plugin UI adapts to dark mode', async ({ page }) => {
Tests run automatically in GitHub Actions on pushes to `main` and pull requests. See `.github/workflows/e2e.yaml` for workflow configuration.
### Required Secrets
### Architecture
Configure these in GitHub repository settings (Settings → Secrets and variables → Actions):
The E2E workflow deploys a **dedicated Headlamp instance** for each test run:
- `HEADLAMP_URL` (optional): Headlamp instance URL
- `AUTHENTIK_USERNAME` + `AUTHENTIK_PASSWORD` (for OIDC auth)
- OR `HEADLAMP_TOKEN` (for token-based auth)
1. Build plugin and Docker image (`Dockerfile.e2e`)
2. Push image to `ghcr.io/privilegedescalation/headlamp-polaris-e2e:<sha>`
3. Deploy via Helm in the `headlamp-e2e` namespace (`scripts/deploy-e2e-headlamp.sh`)
4. Run Playwright tests against the E2E instance
5. Tear down (`scripts/teardown-e2e-headlamp.sh`)
### Workflow Overview
No PVCs, no kubectl exec/cp, no patching of existing deployments. The plugin is baked into the Docker image.
1. Checkout code
2. Setup Node.js 20 with npm cache
3. Install dependencies (`npm ci`)
4. Install Playwright browsers (`chromium` only)
5. Run auth setup (creates session in `e2e/.auth/state.json`)
6. Run all E2E tests
7. Upload artifacts on failure:
- `playwright-report/` - HTML test report
- `test-results/` - Screenshots, traces, videos
### Cluster Prerequisites
One-time setup by a cluster admin:
```bash
kubectl create namespace headlamp-e2e
kubectl apply -f deployment/e2e-ci-runner-rbac.yaml
```
### Manual Trigger
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env bash
# deploy-e2e-headlamp.sh
#
# Builds a custom Headlamp image with the polaris plugin pre-installed,
# pushes it to ghcr.io, and deploys a dedicated E2E Headlamp instance.
#
# This replaces the old PVC + kubectl-patch approach. The plugin is part
# of the container image — no PVCs, no kubectl exec/cp, no deployment
# patching required.
#
# Prerequisites:
# - Plugin built (dist/ exists)
# - Docker or buildx available
# - GHCR_TOKEN set (or GH_TOKEN with packages:write)
# - kubectl configured with cluster access
# - Helm 3 installed
#
# Environment:
# E2E_NAMESPACE — namespace for E2E Headlamp (default: headlamp-e2e)
# E2E_RELEASE — Helm release name (default: headlamp-e2e)
# HEADLAMP_VERSION — base Headlamp image tag (default: latest)
# IMAGE_TAG — tag for the E2E image (default: git SHA)
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
DIST_DIR="$REPO_ROOT/dist"
E2E_NAMESPACE="${E2E_NAMESPACE:-headlamp-e2e}"
E2E_RELEASE="${E2E_RELEASE:-headlamp-e2e}"
HEADLAMP_VERSION="${HEADLAMP_VERSION:-latest}"
IMAGE_REPO="ghcr.io/privilegedescalation/headlamp-polaris-e2e"
IMAGE_TAG="${IMAGE_TAG:-$(git -C "$REPO_ROOT" rev-parse --short HEAD)}"
IMAGE="${IMAGE_REPO}:${IMAGE_TAG}"
if [ ! -d "$DIST_DIR" ]; then
echo "ERROR: dist/ not found. Run 'npm run build' first." >&2
exit 1
fi
echo "=== E2E Headlamp Deployment ==="
echo " Image: $IMAGE"
echo " Namespace: $E2E_NAMESPACE"
echo " Release: $E2E_RELEASE"
# --- Build and push the custom image ---
echo ""
echo "Building E2E Headlamp image..."
docker build -f "$REPO_ROOT/Dockerfile.e2e" \
--build-arg "HEADLAMP_VERSION=${HEADLAMP_VERSION}" \
-t "$IMAGE" \
"$REPO_ROOT"
echo "Pushing image to ghcr.io..."
docker push "$IMAGE"
# --- Deploy with Helm ---
echo ""
echo "Adding Headlamp Helm repo..."
helm repo add headlamp https://headlamp-k8s.github.io/headlamp/ --force-update
helm repo update
echo "Creating namespace ${E2E_NAMESPACE} (if needed)..."
kubectl create namespace "$E2E_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
echo "Installing/upgrading Headlamp E2E instance..."
helm upgrade --install "$E2E_RELEASE" headlamp/headlamp \
-n "$E2E_NAMESPACE" \
-f "$REPO_ROOT/deployment/headlamp-e2e-values.yaml" \
--set "image.registry=ghcr.io" \
--set "image.repository=privilegedescalation/headlamp-polaris-e2e" \
--set "image.tag=${IMAGE_TAG}" \
--wait \
--timeout 120s
echo "Waiting for rollout..."
kubectl rollout status "deployment/${E2E_RELEASE}-headlamp" \
-n "$E2E_NAMESPACE" --timeout=120s
# --- Generate a service URL for tests ---
SVC_URL="http://${E2E_RELEASE}-headlamp.${E2E_NAMESPACE}.svc.cluster.local"
echo ""
echo "E2E Headlamp is ready at: ${SVC_URL}"
echo " export HEADLAMP_URL=${SVC_URL}"
# --- Generate a token for test auth ---
echo ""
echo "Creating service account token for E2E auth..."
kubectl create serviceaccount headlamp-e2e-test \
-n "$E2E_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
TOKEN=$(kubectl create token headlamp-e2e-test -n "$E2E_NAMESPACE" --duration=1h 2>/dev/null || echo "")
if [ -n "$TOKEN" ]; then
echo " export HEADLAMP_TOKEN=<generated>"
echo ""
echo "HEADLAMP_URL=${SVC_URL}" > "$REPO_ROOT/.env.e2e"
echo "HEADLAMP_TOKEN=${TOKEN}" >> "$REPO_ROOT/.env.e2e"
echo "Wrote .env.e2e with HEADLAMP_URL and HEADLAMP_TOKEN"
else
echo " WARNING: Could not generate token. Set HEADLAMP_TOKEN manually or use OIDC."
fi
echo ""
echo "E2E deployment complete."
-135
View File
@@ -1,135 +0,0 @@
#!/usr/bin/env bash
# deploy-plugin-via-volume.sh
#
# Copies the built plugin into the shared PVC so Headlamp picks it up.
# Uses a temporary Kubernetes Job to write to the PVC — the CI runner
# does NOT need the PVC mounted locally.
#
# Usage:
# scripts/deploy-plugin-via-volume.sh
#
# Environment:
# HEADLAMP_NAMESPACE — namespace where Headlamp runs (default: kube-system)
# HEADLAMP_DEPLOY — Headlamp deployment name (default: headlamp)
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
HEADLAMP_NAMESPACE="${HEADLAMP_NAMESPACE:-kube-system}"
HEADLAMP_DEPLOY="${HEADLAMP_DEPLOY:-headlamp}"
# The deployed directory name must match the package.json name and
# the registerPluginSettings name. Headlamp identifies plugins by
# reading package.json from each subdirectory of the plugins dir.
PLUGIN_DIR_NAME="headlamp-polaris"
DIST_DIR="$REPO_ROOT/dist"
if [ ! -d "$DIST_DIR" ]; then
echo "ERROR: dist/ not found. Run 'npm run build' first." >&2
exit 1
fi
echo "Deploying plugin to shared volume via temporary job..."
echo " Source: $DIST_DIR"
echo " PVC: headlamp-plugins"
echo " Plugin: $PLUGIN_DIR_NAME"
# Create tarball of plugin dist + package.json
TAR_FILE=$(mktemp /tmp/plugin-XXXXXX.tar.gz)
tar -czf "$TAR_FILE" -C "$DIST_DIR" . -C "$REPO_ROOT" package.json
echo " Tarball: $TAR_FILE ($(du -h "$TAR_FILE" | cut -f1))"
# Find the node where Headlamp is running — the PVC is ReadWriteOnce so
# the deploy job must land on the same node to mount it.
HEADLAMP_NODE=$(kubectl get pods -n "$HEADLAMP_NAMESPACE" \
-l "app.kubernetes.io/name=headlamp" \
-o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true)
if [ -z "$HEADLAMP_NODE" ]; then
HEADLAMP_NODE=$(kubectl get pods -n "$HEADLAMP_NAMESPACE" \
-l "app.kubernetes.io/instance=headlamp" \
-o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true)
fi
if [ -n "$HEADLAMP_NODE" ]; then
echo " Headlamp node: $HEADLAMP_NODE (scheduling deploy job there)"
fi
# Clean up any previous deploy resources
kubectl delete pod plugin-deploy -n "$HEADLAMP_NAMESPACE" --ignore-not-found --wait=true 2>/dev/null || true
kubectl delete configmap plugin-tarball -n "$HEADLAMP_NAMESPACE" --ignore-not-found 2>/dev/null || true
sleep 2
# Store the tarball in a ConfigMap (binary-safe via --from-file)
echo "Creating ConfigMap with plugin tarball..."
kubectl create configmap plugin-tarball \
-n "$HEADLAMP_NAMESPACE" \
--from-file=plugin.tar.gz="$TAR_FILE"
# Build the Pod manifest as a temp file to avoid heredoc YAML escaping issues
POD_FILE=$(mktemp /tmp/plugin-deploy-pod-XXXXXX.yaml)
cat > "$POD_FILE" <<'YAMLDOC'
apiVersion: v1
kind: Pod
metadata:
name: plugin-deploy
spec:
restartPolicy: Never
containers:
- name: deploy
image: busybox:1.36
command: ["sh", "-c"]
args:
- |
echo "Cleaning up stale plugin directories..."
rm -rf /plugins/polaris /plugins/headlamp-polaris
echo "Extracting plugin to shared volume..."
mkdir -p /plugins/PLUGIN_DIR_PLACEHOLDER
tar -xzf /tarball/plugin.tar.gz -C /plugins/PLUGIN_DIR_PLACEHOLDER
echo "Files deployed:"
ls -la /plugins/PLUGIN_DIR_PLACEHOLDER/
volumeMounts:
- name: plugins
mountPath: /plugins
- name: tarball
mountPath: /tarball
readOnly: true
volumes:
- name: plugins
persistentVolumeClaim:
claimName: headlamp-plugins
- name: tarball
configMap:
name: plugin-tarball
YAMLDOC
# Substitute plugin dir name
sed -i "s/PLUGIN_DIR_PLACEHOLDER/${PLUGIN_DIR_NAME}/g" "$POD_FILE"
# Add nodeName if we know which node Headlamp is on
if [ -n "$HEADLAMP_NODE" ]; then
sed -i "/restartPolicy: Never/i\\ nodeName: ${HEADLAMP_NODE}" "$POD_FILE"
fi
echo "Starting deploy pod..."
kubectl apply -n "$HEADLAMP_NAMESPACE" -f "$POD_FILE"
rm -f "$POD_FILE"
# Wait for the pod to complete (Succeeded phase)
echo "Waiting for deploy pod to complete..."
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/plugin-deploy \
-n "$HEADLAMP_NAMESPACE" --timeout=120s
# Show logs
kubectl logs plugin-deploy -n "$HEADLAMP_NAMESPACE" 2>/dev/null || true
# Clean up
kubectl delete pod plugin-deploy -n "$HEADLAMP_NAMESPACE" --ignore-not-found 2>/dev/null || true
kubectl delete configmap plugin-tarball -n "$HEADLAMP_NAMESPACE" --ignore-not-found 2>/dev/null || true
rm -f "$TAR_FILE"
# Restart Headlamp to pick up the new plugin
echo "Restarting Headlamp deployment to load plugin..."
kubectl rollout restart "deployment/$HEADLAMP_DEPLOY" -n "$HEADLAMP_NAMESPACE"
kubectl rollout status "deployment/$HEADLAMP_DEPLOY" -n "$HEADLAMP_NAMESPACE" --timeout=120s
echo "Plugin deployed successfully."
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# teardown-e2e-headlamp.sh
#
# Tears down the dedicated E2E Headlamp instance deployed by deploy-e2e-headlamp.sh.
#
# Environment:
# E2E_NAMESPACE — namespace to clean up (default: headlamp-e2e)
# E2E_RELEASE — Helm release to uninstall (default: headlamp-e2e)
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
E2E_NAMESPACE="${E2E_NAMESPACE:-headlamp-e2e}"
E2E_RELEASE="${E2E_RELEASE:-headlamp-e2e}"
echo "=== E2E Headlamp Teardown ==="
echo " Namespace: $E2E_NAMESPACE"
echo " Release: $E2E_RELEASE"
echo "Uninstalling Helm release..."
helm uninstall "$E2E_RELEASE" -n "$E2E_NAMESPACE" 2>/dev/null || echo "Release not found (already removed?)"
echo "Deleting namespace..."
kubectl delete namespace "$E2E_NAMESPACE" --ignore-not-found --wait=false
# Clean up local env file
rm -f "$REPO_ROOT/.env.e2e"
echo "Teardown complete."