headlamp-polaris-plugin/docs/TROUBLESHOOTING.md

Troubleshooting Guide

This guide covers common issues encountered when using the Headlamp Polaris Plugin and their solutions.

Table of Contents

• Plugin Not Showing in Sidebar
• 403 Forbidden Error
• 404 Not Found Error
• Plugin Settings Page Empty
• Dark Mode Issues
• Data Not Loading / Infinite Spinner
• Browser Console Errors
• Network and RBAC Debugging
• Plugin Installation Issues
• ArtifactHub Sync Delays

---

Plugin Not Showing in Sidebar

Symptoms

• Plugin appears in Settings → Plugins but sidebar entry is missing
• No "Polaris" section in navigation
• Routes like /polaris return 404 or blank page

Common Causes

1. Headlamp v0.39.0+ Plugin Loading Issue

Root Cause: Headlamp v0.39.0+ changed plugin loading behavior. With config.watchPlugins: true (default), catalog-managed plugins are treated as "development directory" plugins, causing the backend to serve metadata but frontend to never execute the JavaScript.

Solution: Set config.watchPlugins: false in Headlamp configuration.

# HelmRelease values
config:
  watchPlugins: false  # CRITICAL for plugin manager

After applying this change:

1. Restart Headlamp pod
2. Hard refresh browser (Cmd+Shift+R or Ctrl+Shift+R)
3. Clear browser cache if needed

References: See deployment/PLUGIN_LOADING_FIX.md for complete root cause analysis.

2. Plugin Not Installed

Check plugin installation:

# View Headlamp pod logs (plugin sidecar)
kubectl logs -n kube-system deployment/headlamp -c headlamp-plugin

# Expected output:
# Installing plugin from https://github.com/.../headlamp-polaris-plugin-X.Y.Z.tar.gz
# Plugin installed successfully

Verify plugin files exist:

kubectl exec -n kube-system deployment/headlamp -c headlamp -- ls -la /headlamp/plugins/
# Should show: headlamp-polaris-plugin/

3. JavaScript Cached by Browser

After upgrading the plugin, old JavaScript may be cached.

Solution:

• Hard refresh: Cmd+Shift+R (macOS) or Ctrl+Shift+R (Linux/Windows)
• Clear browser cache for Headlamp domain
• Open DevTools → Application → Clear Storage → Clear all

4. Plugin Disabled in Settings

Check Settings → Plugins:

• Navigate to Headlamp Settings → Plugins
• Ensure "Polaris" plugin is enabled (toggle should be ON)
• If disabled, enable it and refresh the page

---

403 Forbidden Error

Symptoms

• Error message: "Error loading Polaris audit data: 403 Forbidden"
• Browser console shows 403 response from API proxy
• Plugin sidebar shows but data fails to load

Root Cause

User or service account lacks services/proxy permission on polaris-dashboard service in the polaris namespace.

Solution

1. Verify RBAC Configuration

Check if Role exists:

kubectl get role polaris-proxy-reader -n polaris -o yaml

Expected output:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: polaris-proxy-reader
  namespace: polaris
rules:
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["polaris-dashboard"]
    verbs: ["get"]

2. Verify RoleBinding

For service account mode:

kubectl get rolebinding headlamp-polaris-proxy -n polaris -o yaml

Expected subjects:

subjects:
  - kind: ServiceAccount
    name: headlamp
    namespace: kube-system

For OIDC mode:

kubectl get rolebinding -n polaris -o yaml | grep -A 5 polaris-proxy-reader

Ensure your user or group is bound to the polaris-proxy-reader role.

3. Create Missing RBAC

If RBAC is missing, apply the minimal configuration:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: polaris-proxy-reader
  namespace: polaris
rules:
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["polaris-dashboard"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: headlamp-polaris-proxy
  namespace: polaris
subjects:
  - kind: ServiceAccount
    name: headlamp
    namespace: kube-system
roleRef:
  kind: Role
  name: polaris-proxy-reader
  apiGroup: rbac.authorization.k8s.io
EOF

4. Test RBAC Permissions

Service account mode:

# Impersonate Headlamp service account
kubectl auth can-i get services/proxy \
  --as=system:serviceaccount:kube-system:headlamp \
  --resource-name=polaris-dashboard \
  -n polaris
# Expected: yes

OIDC mode (test as yourself):

kubectl auth can-i get services/proxy \
  --resource-name=polaris-dashboard \
  -n polaris
# Expected: yes (if you have proper RoleBinding)

5. Restart Headlamp

After applying RBAC changes:

kubectl rollout restart deployment headlamp -n kube-system

---

404 Not Found Error

Symptoms

• Error message: "Error loading Polaris audit data: 404 Not Found"
• Service proxy request returns 404
• Polaris dashboard not reachable

Root Cause

Polaris dashboard service doesn't exist or is in a different namespace.

Solution

1. Verify Polaris Installation

Check if Polaris is installed:

kubectl get pods -n polaris
# Expected: polaris-dashboard-* pod running

Check if service exists:

kubectl get service polaris-dashboard -n polaris
# Expected: ClusterIP service on port 80

2. Verify Service Name and Port

The plugin expects:

• Namespace: polaris
• Service Name: polaris-dashboard
• Port: 80 (or named port dashboard)

If your service has a different name or is in a different namespace, you'll need to modify the plugin source or redeploy Polaris with standard naming.

3. Test Service Proxy Manually

kubectl proxy &
curl http://localhost:8001/api/v1/namespaces/polaris/services/polaris-dashboard:80/proxy/results.json

If this returns JSON, the service proxy works and the issue is elsewhere. If this returns 404, Polaris service is not configured correctly.

4. Check Polaris Dashboard Configuration

Verify Polaris is running with dashboard enabled:

kubectl get deployment polaris-dashboard -n polaris -o yaml | grep -A 5 dashboard

If dashboard.enabled: false in Helm values, enable it:

# values.yaml
dashboard:
  enabled: true

5. Reinstall Polaris

If Polaris is missing or misconfigured:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm upgrade --install polaris fairwinds-stable/polaris \
  --namespace polaris \
  --create-namespace \
  --set dashboard.enabled=true

---

Plugin Settings Page Empty

Symptoms

• Settings → Polaris shows title but no content
• Refresh interval and dashboard URL fields not visible

Root Cause (Fixed in v0.3.3)

Plugin settings registration name didn't match package.json name.

Solution

Upgrade to v0.3.3 or later:

# Via Headlamp UI: Settings → Plugins → Update
# Or redeploy with latest version

If manually installing, ensure plugin name matches package.json:

registerPluginSettings('headlamp-polaris-plugin', PolarisSettings, true);
// NOT 'polaris' — must match package.json name

---

Dark Mode Issues

Symptoms

• Drawer background remains white in dark mode
• Text is hard to read in dark mode
• Theme colors don't match Headlamp UI

Solution (Fixed in v0.3.5)

Upgrade to v0.3.5 or later for complete dark mode support.

Verify CSS Variables: The plugin uses MUI CSS variables for theming:

• --mui-palette-background-default (drawer background)
• --mui-palette-text-primary (text color)
• --mui-palette-primary-main (links, buttons)
• --mui-palette-error-main (danger states)

These automatically adapt to Headlamp's theme (light/dark/system).

Hard Refresh Required: After upgrading from v0.3.4 or earlier, hard refresh your browser:

• macOS: Cmd+Shift+R
• Linux/Windows: Ctrl+Shift+R

Clear Browser Cache: If hard refresh doesn't help, clear cache for Headlamp domain.

---

Data Not Loading / Infinite Spinner

Symptoms

• Plugin shows "Loading Polaris audit data..." forever
• No error message in UI
• Data never appears

Debugging Steps

1. Check Browser Console

Open DevTools (F12) → Console tab.

Look for:

• Network errors (CORS, timeouts, 5xx responses)
• JavaScript errors
• Failed API requests

2. Check Network Tab

Open DevTools → Network tab → Filter by "results.json"

Expected request:

GET /api/v1/namespaces/polaris/services/polaris-dashboard:80/proxy/results.json
Status: 200
Response: JSON data

Common issues:

• Status 0 / Failed: Network policy blocking request
• Status 403: RBAC issue (see 403 Forbidden Error)
• Status 404: Service not found (see 404 Not Found Error)
• Status 500: Polaris dashboard error
• Status 502/504: Service unreachable (network policy or pod down)

3. Check Polaris Dashboard Health

# Check if Polaris pod is running
kubectl get pods -n polaris

# Check Polaris logs
kubectl logs -n polaris deployment/polaris-dashboard

# Test direct access to Polaris
kubectl port-forward -n polaris svc/polaris-dashboard 8080:80
curl http://localhost:8080/results.json

4. Check Network Policies

If your cluster uses NetworkPolicies:

kubectl get networkpolicy -n polaris

Ensure API server (or Headlamp pod) can reach Polaris dashboard.

Example fix:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-server-to-polaris
  namespace: polaris
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: polaris
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector: {}  # Allow from all namespaces (API server)
      ports:
        - protocol: TCP
          port: 8080

5. Increase Timeout / Disable Auto-Refresh

If Polaris responds slowly:

• Open Settings → Polaris
• Increase refresh interval to 10+ minutes
• Or set to "Manual only" to disable auto-refresh

---

Browser Console Errors

Common Errors and Solutions

Error: "Failed to fetch"

Cause: Network request failed (CORS, network policy, timeout)

Solution:

1. Check Network tab for actual HTTP status
2. Verify network policies allow API server → Polaris
3. Check Polaris pod is running

---

Error: "Unexpected token < in JSON"

Cause: API returned HTML (error page) instead of JSON

Solution:

1. Check Network tab response body (likely 404 or 500 error page)
2. Verify Polaris service exists and is healthy
3. Check service proxy URL is correct

---

Error: "registerPluginSettings is not a function"

Cause: Headlamp version too old (< v0.26)

Solution: Upgrade Headlamp to v0.26 or later.

---

Error: "Cannot read property 'AuditData' of undefined"

Cause: Polaris returned empty or malformed response

Solution:

1. Check Polaris logs for errors
2. Verify Polaris is scanning the cluster (check audit timestamp)
3. Test /results.json endpoint directly

---

Network and RBAC Debugging

Comprehensive RBAC Test

Run this script to test all RBAC components:

#!/bin/bash
NS="polaris"
SA="headlamp"
SA_NS="kube-system"

echo "=== Testing RBAC for Polaris Plugin ==="

# 1. Check if service exists
echo "1. Service check:"
kubectl get svc polaris-dashboard -n $NS || echo "❌ Service not found"

# 2. Check if Role exists
echo "2. Role check:"
kubectl get role polaris-proxy-reader -n $NS || echo "❌ Role not found"

# 3. Check if RoleBinding exists
echo "3. RoleBinding check:"
kubectl get rolebinding headlamp-polaris-proxy -n $NS || echo "❌ RoleBinding not found"

# 4. Test service account permissions
echo "4. Permission test (service account):"
kubectl auth can-i get services/proxy \
  --as=system:serviceaccount:$SA_NS:$SA \
  --resource-name=polaris-dashboard \
  -n $NS

# 5. Test actual proxy request (requires kubectl proxy)
echo "5. Proxy test:"
kubectl proxy &
PROXY_PID=$!
sleep 2
curl -s http://localhost:8001/api/v1/namespaces/$NS/services/polaris-dashboard:80/proxy/results.json | jq '.DisplayName' || echo "❌ Proxy request failed"
kill $PROXY_PID

echo "=== Test complete ==="

Network Policy Debugging

Test connectivity from Headlamp to Polaris:

# Create debug pod in kube-system namespace
kubectl run netdebug -n kube-system --rm -it --image=nicolaka/netshoot -- bash

# Inside pod, test DNS and HTTP
nslookup polaris-dashboard.polaris.svc.cluster.local
curl -v http://polaris-dashboard.polaris.svc.cluster.local/results.json

If this fails, network policies are blocking traffic.

API Server Audit Logs

If you have audit logging enabled, check for denied requests:

# View recent audit logs (location varies by cluster)
kubectl logs -n kube-system kube-apiserver-* | grep polaris-dashboard

# Look for lines with:
# "reason": "Forbidden"
# "user": "system:serviceaccount:kube-system:headlamp"

---

Plugin Installation Issues

Sidecar Fails to Install Plugin

Symptoms:

• Plugin sidecar logs show download errors
• Plugin directory is empty
• Settings → Plugins shows nothing

Check sidecar logs:

kubectl logs -n kube-system deployment/headlamp -c headlamp-plugin

Common errors:

1. Network timeout downloading tarball

Error: connect ETIMEDOUT

Solution: Check cluster egress network policies allow HTTPS to GitHub.

---

2. Invalid tarball URL

Error: 404 Not Found

Solution: Verify archive-url in plugin config matches GitHub release:

kubectl get configmap headlamp-plugin-config -n kube-system -o yaml

Expected format:

https://github.com/privilegedescalation/headlamp-polaris-plugin/releases/download/v0.3.10/polaris-0.3.10.tar.gz

---

3. Permission denied writing to /headlamp/plugins

Solution: Ensure volume mount is writable:

volumeMounts:
  - name: plugins
    mountPath: /headlamp/plugins

---

Plugin Manager Not Working

Symptoms:

• Headlamp → Settings → Plugins shows "Catalog" tab but plugins don't install
• "Install" button does nothing

Root Cause: Plugin manager requires config.pluginsDir to be set.

Solution: Configure Headlamp for plugin manager:

# HelmRelease values
config:
  pluginsDir: /headlamp/plugins
  watchPlugins: false  # CRITICAL for v0.39.0+

---

ArtifactHub Sync Delays

Symptoms

• New version released on GitHub but not showing in ArtifactHub
• Headlamp plugin catalog shows old version

Root Cause

ArtifactHub pulls metadata every 30 minutes. There is no webhook or push mechanism.

Solution

Wait 30 minutes after pushing a GitHub release, then check:

https://artifacthub.io/packages/headlamp/headlamp-polaris-plugin/headlamp-polaris-plugin

Verify metadata:

1. Check artifacthub-pkg.yml is in repository root
2. Check headlamp/plugin/archive-url points to GitHub release
3. Check headlamp/plugin/archive-checksum matches tarball SHA256

Force sync (ArtifactHub UI):

• Log in to ArtifactHub as package maintainer
• Go to package settings
• Click "Reindex now"

Note: First sync after repository registration may take up to 1 hour.

---

Still Having Issues?

If none of these solutions work, gather debugging information and open an issue:

Required Information

1. Version Information:

   kubectl get pods -n kube-system -l app.kubernetes.io/name=headlamp -o yaml | grep image:
   

2. Plugin Version:

   • Check Settings → Plugins in Headlamp UI
   • Or: kubectl exec -n kube-system deployment/headlamp -c headlamp -- cat /headlamp/plugins/headlamp-polaris-plugin/package.json

3. Browser Console Output:

   • Open DevTools (F12) → Console
   • Screenshot or copy errors

4. Network Tab:

   • Open DevTools → Network
   • Screenshot failed requests to results.json

5. Pod Logs:

   kubectl logs -n kube-system deployment/headlamp -c headlamp --tail=100
   kubectl logs -n polaris deployment/polaris-dashboard --tail=100
   

6. RBAC Configuration:

   kubectl get role,rolebinding -n polaris
   

Where to Get Help

• GitHub Issues: https://github.com/privilegedescalation/headlamp-polaris-plugin/issues
• GitHub Discussions: https://github.com/privilegedescalation/headlamp-polaris-plugin/discussions

Include the debugging information above when opening an issue.