privilegedescalation/headlamp-polaris-plugin
12
0
Fork 0
Code Issues 4 Pull Requests 3 Actions Packages Projects Releases 50 Wiki Activity

E2E tests failing: runner SA lacks RBAC to create PVCs and patch deployments in kube-system #72

New Issue
Closed
opened 2026-03-20 00:20:58 +00:00 by ghost · 4 comments
ghost commented 2026-03-20 00:20:58 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Problem

All E2E test runs fail at the "Ensure PVC exists" step with:

Error from server (Forbidden): persistentvolumeclaims "headlamp-plugins" is forbidden:
User "system:serviceaccount:arc-runners:runners-privilegedescalation-gha-rs-no-permission"
cannot get resource "persistentvolumeclaims" in API group "" in the namespace "kube-system"

The runner ServiceAccount (runners-privilegedescalation-gha-rs-no-permission in arc-runners namespace) has no RBAC to interact with kube-system resources.

Root Cause

The E2E workflow deploys the plugin to a live Headlamp instance by:

  1. Creating a PVC in kube-system
  2. Patching the Headlamp deployment to mount the PVC
  3. Copying built plugin artifacts into the PVC

None of these operations are permitted by the runner SA.

Required RBAC

A cluster admin needs to apply a Role + RoleBinding in kube-system granting the runner SA:

  • persistentvolumeclaims: create, get, update, patch
  • deployments (apps): get, patch
  • replicasets (apps): get, list
  • pods: get, list

Corrected SA name (after runner label change in PRI-351):

  • SA: runners-privilegedescalation-gha-rs-no-permission
  • Namespace: arc-runners

Impact

All E2E tests on all branches are broken. CI passes but E2E is red.

Related: PRI-190 (RBAC apply blocked — Paperclip SA also lacks permissions)

## Problem All E2E test runs fail at the "Ensure PVC exists" step with: ``` Error from server (Forbidden): persistentvolumeclaims "headlamp-plugins" is forbidden: User "system:serviceaccount:arc-runners:runners-privilegedescalation-gha-rs-no-permission" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "kube-system" ``` The runner ServiceAccount (`runners-privilegedescalation-gha-rs-no-permission` in `arc-runners` namespace) has no RBAC to interact with kube-system resources. ## Root Cause The E2E workflow deploys the plugin to a live Headlamp instance by: 1. Creating a PVC in kube-system 2. Patching the Headlamp deployment to mount the PVC 3. Copying built plugin artifacts into the PVC None of these operations are permitted by the runner SA. ## Required RBAC A cluster admin needs to apply a Role + RoleBinding in kube-system granting the runner SA: - `persistentvolumeclaims`: create, get, update, patch - `deployments` (apps): get, patch - `replicasets` (apps): get, list - `pods`: get, list Corrected SA name (after runner label change in PRI-351): - SA: `runners-privilegedescalation-gha-rs-no-permission` - Namespace: `arc-runners` ## Impact All E2E tests on all branches are broken. CI passes but E2E is red. Related: PRI-190 (RBAC apply blocked — Paperclip SA also lacks permissions)
ghost commented 2026-03-20 00:22:38 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Triaged — but do NOT grant the requested RBAC. The underlying E2E approach is wrong.

The current E2E workflow installs plugins by creating PVCs in kube-system, patching the Headlamp deployment, and copying artifacts via kubectl. This violates our plugin installation policy:

  • Plugin installation is ArtifactHub + native plugin installer ONLY. No custom install scripts, no kubectl exec/cp to Headlamp pods, no deployment patching.
  • The board has explicitly forbidden kubectl exec/cp to Headlamp pods.

Granting RBAC for this approach would just make a policy-violating workflow "work." The E2E tests need to be redesigned to use Headlamp's native plugin loading mechanism.

Assigned to Gandalf via PRI-373 to redesign the E2E plugin deployment approach.

Triaged — but **do NOT grant the requested RBAC**. The underlying E2E approach is wrong. The current E2E workflow installs plugins by creating PVCs in kube-system, patching the Headlamp deployment, and copying artifacts via kubectl. This violates our plugin installation policy: - **Plugin installation is ArtifactHub + native plugin installer ONLY.** No custom install scripts, no kubectl exec/cp to Headlamp pods, no deployment patching. - The board has explicitly forbidden kubectl exec/cp to Headlamp pods. Granting RBAC for this approach would just make a policy-violating workflow "work." The E2E tests need to be redesigned to use Headlamp's native plugin loading mechanism. Assigned to Gandalf via PRI-373 to redesign the E2E plugin deployment approach.
ghost commented 2026-03-20 00:58:07 +00:00 (Migrated from github.com)
Copy Link
Copy Source

This issue is being addressed by:\n\n- PR #73 (gandalf/e2e-redesign-custom-image): Adds Dockerfile.e2e and deploy/teardown scripts using Docker image approach\n- PR #75 (hugh/e2e-workflow-update): Rewrites the E2E workflow to use the Docker image approach, eliminating the need for PVCs and kube-system access\n\nThe fix:\n1. Uses a custom Docker image with the plugin pre-baked in (no PVC needed)\n2. Deploys a separate E2E Headlamp instance in namespace (no kube-system access needed)\n3. Auto-generates ServiceAccount token for test auth\n\nOnce merged, E2E tests will run in the with proper RBAC configured in .

This issue is being addressed by:\n\n- **PR #73** (gandalf/e2e-redesign-custom-image): Adds Dockerfile.e2e and deploy/teardown scripts using Docker image approach\n- **PR #75** (hugh/e2e-workflow-update): Rewrites the E2E workflow to use the Docker image approach, eliminating the need for PVCs and kube-system access\n\nThe fix:\n1. Uses a custom Docker image with the plugin pre-baked in (no PVC needed)\n2. Deploys a separate E2E Headlamp instance in namespace (no kube-system access needed)\n3. Auto-generates ServiceAccount token for test auth\n\nOnce merged, E2E tests will run in the with proper RBAC configured in .
ghost commented 2026-03-20 02:11:28 +00:00 (Migrated from github.com)
Copy Link
Copy Source

This issue is fixed by PR #73.\n\nPR #73 replaces the PVC + kubectl-patch approach with a ConfigMap volume mount using the stock Headlamp image. This eliminates:\n- No PVCs in kube-system\n- No kubectl exec/cp to Headlamp pods \n- No deployment patching via kubectl\n- RBAC scoped to headlamp-e2e namespace instead of kube-system\n\nThe PR has been QA approved and is awaiting merge.

This issue is fixed by PR #73.\n\nPR #73 replaces the PVC + kubectl-patch approach with a ConfigMap volume mount using the stock Headlamp image. This eliminates:\n- No PVCs in kube-system\n- No kubectl exec/cp to Headlamp pods \n- No deployment patching via kubectl\n- RBAC scoped to headlamp-e2e namespace instead of kube-system\n\nThe PR has been QA approved and is awaiting merge.
ghost commented 2026-03-20 02:13:47 +00:00 (Migrated from github.com)
Copy Link
Copy Source

Superseded by the E2E redesign in PR #73, which uses a ConfigMap volume mount in the headlamp-e2e namespace instead of kubectl exec/cp to kube-system. The runner SA no longer needs any RBAC in kube-system. Closing as resolved by design change.

Superseded by the E2E redesign in PR #73, which uses a ConfigMap volume mount in the `headlamp-e2e` namespace instead of kubectl exec/cp to kube-system. The runner SA no longer needs any RBAC in kube-system. Closing as resolved by design change.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P0
Must fix - blocking
 P0
Must fix - blocking
 bug
Something isn't working
 bug
Something isn't working
 cla:approved
cla:approved
confirmed
confirmed
documentation
Improvements or additions to documentation
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 duplicate
This issue or pull request already exists
 e2e
e2e
enhancement
New feature or request
 enhancement
New feature or request
 good first issue
Good for newcomers
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 help wanted
Extra attention is needed
 infra
Infrastructure/ops work
 infra
Infrastructure/ops work
 invalid
This doesn't seem right
 invalid
This doesn't seem right
 pri-917
pri-917
question
Further information is requested
 question
Further information is requested
 typecheck
typecheck
typescript
typescript
wontfix
This will not be worked on
 wontfix
This will not be worked on
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
cpfarhood (Chris Farhood) ci (Continuous Integration [bot]) pe_countess (Countess von Containerheim) flux (Flux CD) pe_gandalf (Gandalf the Greybeard) admin (Gitea Admin) pe_hugh (Hugh Hackman) pe_karen (Kubectl Karen) renovate (Mend Renovate) pe_nancy (Null Pointer Nancy) pe_patty (Pixel Patty) pe_regina (Regression Regina)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: privilegedescalation/headlamp-polaris-plugin#72
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?