headlamp-sealed-secrets-plugin/docs/troubleshooting/encryption-failures.md

Encryption Failures

Debugging encryption errors when creating sealed secrets.

Table of Contents

• Understanding the Encryption Flow
• Certificate Problems
• Value Issues
• Scope Problems
• Browser Issues
• Network Issues

---

Understanding the Encryption Flow

Before troubleshooting, understand how encryption works:

1. Plugin fetches public certificate from controller
   GET /api/v1/namespaces/kube-system/services/sealed-secrets-controller:http/proxy/v1/cert.pem

2. Plugin validates certificate (PEM format, expiry, fingerprint)

3. Plugin encrypts value client-side using RSA-OAEP
   - Generates random AES-256 key
   - Encrypts value with AES-256-GCM
   - Encrypts AES key with RSA public key
   - Combines into encrypted payload

4. Plugin creates SealedSecret with encrypted data
   POST /apis/bitnami.com/v1alpha1/namespaces/<ns>/sealedsecrets

5. Controller unseals secret server-side

Key Points:

• Encryption happens in browser (plaintext never leaves your machine)
• Certificate must be valid PEM format and not expired
• Maximum value size: ~1MB (RSA limitation)

---

Certificate Problems

"Failed to fetch certificate"

Symptom

Failed to fetch certificate: Network error

Diagnosis

# 1. Check controller is running
kubectl get pods -n kube-system -l name=sealed-secrets-controller

# 2. Test certificate endpoint directly
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
# In another terminal:
curl http://localhost:8080/v1/cert.pem

Solutions

Controller not running: See Controller Issues

Certificate endpoint not responding:

# Check controller logs
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=50

# Restart controller
kubectl rollout restart deployment -n kube-system sealed-secrets-controller

RBAC permission denied:

# Check service access permission
kubectl auth can-i get services/sealed-secrets-controller -n kube-system

# If no, apply RBAC (requires cluster-admin):
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sealed-secrets-cert-downloader
rules:
- apiGroups: [""]
  resources: ["services", "services/proxy"]
  verbs: ["get"]
  resourceNames: ["sealed-secrets-controller"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: $(kubectl config view --minify -o jsonpath='{.contexts[0].context.user}')-cert-downloader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sealed-secrets-cert-downloader
subjects:
- kind: User
  name: $(kubectl config view --minify -o jsonpath='{.contexts[0].context.user}')
  apiGroup: rbac.authorization.k8s.io
EOF

---

"Certificate expired"

Symptom

Encryption failed: Certificate expired on 2025-01-15T10:30:00Z

Diagnosis

# Check certificate expiry
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | \
  openssl x509 -noout -dates

# Example output:
# notBefore=Jan  1 00:00:00 2025 GMT
# notAfter=Jan 15 10:30:00 2025 GMT  ← Expired!

Solution

Rotate sealing keys (see Secret Rotation Tutorial):

# Option 1: Delete old key (generates new automatically)
kubectl delete secret -n kube-system sealed-secrets-key
kubectl rollout restart deployment -n kube-system sealed-secrets-controller

# Option 2: Annotate for rotation (keeps old for decryption)
kubectl annotate secret -n kube-system sealed-secrets-key \
  sealedsecrets.bitnami.com/sealed-secrets-key-rotation=rotate
kubectl rollout restart deployment -n kube-system sealed-secrets-controller

# Wait for new key
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s

# Verify new certificate
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

Warning: After key rotation:

• ✅ Existing SealedSecrets continue to work (controller keeps old key for decryption)
• ❌ Cannot modify existing SealedSecrets (must delete and recreate with new key)

---

"Invalid PEM format"

Symptom

Encryption failed: Certificate is not valid PEM format

Diagnosis

# Fetch and validate certificate
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d > cert.pem

# Should start with:
# -----BEGIN CERTIFICATE-----
# Should end with:
# -----END CERTIFICATE-----

cat cert.pem

Solutions

Corrupted certificate:

# Regenerate certificate
kubectl delete secret -n kube-system sealed-secrets-key
kubectl rollout restart deployment -n kube-system sealed-secrets-controller

Wrong secret: Ensure you're using correct secret:

# List all sealing keys
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

# Should show sealed-secrets-key

---

"Certificate expiring soon" Warning

Symptom

⚠️ Warning: Certificate expires in 15 days

This is an advance warning, not an error. Plugin warns 30 days before expiry.

Action

Plan key rotation before expiry:

1. Schedule maintenance window
2. Backup existing keys:

   kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-keys-backup.yaml
   

3. Rotate keys: See Secret Rotation Tutorial
4. Recreate SealedSecrets if needed

---

Value Issues

"Value too large"

Symptom

Encryption failed: Value exceeds maximum size (1MB)

Cause

RSA encryption has size limit (~1MB for RSA-2048).

Solutions

Split into multiple secrets:

# Instead of:
apiVersion: v1
kind: Secret
metadata:
  name: large-config
data:
  config.json: <2MB base64 data>  ❌

# Use:
---
apiVersion: v1
kind: Secret
metadata:
  name: config-part1
data:
  config-part1.json: <500KB base64 data>  ✅
---
apiVersion: v1
kind: Secret
metadata:
  name: config-part2
data:
  config-part2.json: <500KB base64 data>  ✅

Use external secret management for very large files:

• HashiCorp Vault
• AWS Secrets Manager
• Azure Key Vault
• GCP Secret Manager

Then store only references in SealedSecrets:

apiVersion: v1
kind: Secret
metadata:
  name: external-ref
stringData:
  vault-path: "secret/data/myapp/large-config"

Compress data before encrypting:

# Compress config file
gzip -c config.json | base64

# Then use compressed value in plugin

---

"Empty value"

Symptom

Validation failed: Secret value cannot be empty

Cause

Trying to create secret with empty value.

Solutions

Provide a value:

stringData:
  password: "mysecretvalue"  ✅

Use placeholder if value unknown:

stringData:
  password: "changeme"  # Replace later

Remove key if not needed: Don't create keys with empty values.

---

"Invalid characters"

Symptom

Encryption failed: Value contains invalid characters

This error is rare (plugin accepts any binary data). If you see it:

Solutions

Base64-encode first (for binary data):

# Encode binary file
base64 < binary-file.bin

# Use in plugin with base64-encoded value

Check for null bytes in text:

# Remove null bytes
tr -d '\0' < file.txt

---

Scope Problems

"Scope validation failed"

Symptom

Validation failed: Invalid scope for namespace-scoped secret

Cause

Scope mismatch between plugin setting and cluster policy.

Understanding Scopes

Scope	Can Rename?	Can Move Namespace?	Use Case
strict	❌ No	❌ No	Production secrets
namespace-wide	✅ Yes	❌ No	Dev environments
cluster-wide	✅ Yes	✅ Yes	Shared configs

Solution

Use strict scope (most secure):

# In plugin:
Scope: strict
Name: my-secret
Namespace: production

Match cluster policy: If cluster enforces specific scope, use that:

# Check cluster policy
kubectl get sealedsecrets.bitnami.com -o yaml | grep -A 5 scope

---

"Scope annotation missing"

Symptom

SealedSecret created but controller doesn't unseal it.

Diagnosis

# Check SealedSecret
kubectl get sealedsecret my-secret -n default -o yaml

# Look for annotation:
metadata:
  annotations:
    sealedsecrets.bitnami.com/scope: strict  ← Should be present

Solution

Plugin automatically adds scope annotation. If missing:

1. Delete SealedSecret:

   kubectl delete sealedsecret my-secret -n default
   

2. Recreate with plugin (don't create manually)

---

Browser Issues

"Crypto API not available"

Symptom

Encryption failed: Web Crypto API not available

Cause

Browser doesn't support Web Crypto API (very old browser).

Solution

Upgrade browser:

• Chrome 37+
• Firefox 34+
• Safari 11+
• Edge 79+

Use HTTPS: Web Crypto requires HTTPS (or localhost):

# If using custom Headlamp deployment, enable HTTPS

---

"Encryption timeout"

Symptom

Encryption failed: Operation timeout after 30000ms

Cause

Browser crypto operation taking too long (rare).

Solutions

Reduce value size: If encrypting large value, split it.

Close other tabs: Free up browser resources.

Try different browser: Some browsers have better crypto performance.

Check browser console for JavaScript errors:

1. View → Toggle Developer Tools
2. Console tab
3. Look for errors during encryption

---

Network Issues

"Certificate fetch timeout"

Symptom

Failed to fetch certificate: Request timeout after 30000ms

Diagnosis

# Test cluster connectivity
kubectl cluster-info

# Test service endpoint
kubectl get svc -n kube-system sealed-secrets-controller

# Test with curl
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
curl -m 5 http://localhost:8080/v1/cert.pem

Solutions

Slow network: Increase timeout in plugin settings:

• Settings → Sealed Secrets → Network Timeout: 60s

VPN disconnected: Reconnect VPN if using one.

Firewall blocking: Check firewall allows Kubernetes API access.

---

"CORS error"

Symptom (Browser Console)

Access blocked by CORS policy

Cause

Cross-origin request blocked (should not happen in normal usage).

Solution

Use Headlamp's built-in proxy (enabled by default).

If developing plugin locally:

# Start Headlamp in dev mode
npm start

# Headlamp automatically proxies requests

---

Debugging Encryption

Enable Debug Logging

In browser console:

// Enable debug logs
localStorage.setItem('debug', 'headlamp-sealed-secrets:*')

// Reload page
location.reload()

// Try encryption again
// Check console for detailed logs

Manual Encryption Test

Test encryption manually:

// In browser console
const cert = await fetch('/api/v1/namespaces/kube-system/services/sealed-secrets-controller:http/proxy/v1/cert.pem')
  .then(r => r.text());

console.log('Certificate:', cert);

// Should show PEM certificate starting with:
// -----BEGIN CERTIFICATE-----

Verify Encrypted Output

# After creating SealedSecret, verify it
kubectl get sealedsecret my-secret -n default -o yaml

# Should show:
# spec:
#   encryptedData:
#     password: AgB... (base64 encrypted value)

---

Advanced Debugging

Test with kubeseal CLI

If plugin fails but kubeseal works, it's a plugin issue:

# Install kubeseal
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar -xzf kubeseal-0.24.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

# Test encryption
echo -n mysecretvalue | kubeseal \
  --controller-namespace=kube-system \
  --controller-name=sealed-secrets-controller \
  --format=yaml \
  --name=my-secret \
  --namespace=default \
  --scope=strict

# If this works, plugin has a bug
# If this fails too, controller/cluster issue

Compare Encryption

# Encrypt same value with both methods
# Plugin output:
kubectl get sealedsecret my-secret-plugin -n default -o jsonpath='{.spec.encryptedData.password}'

# kubeseal output:
echo -n mysecretvalue | kubeseal --raw --cert cert.pem --scope strict --name my-secret --namespace default

# Values should be different (encryption is random)
# But both should unseal to same plaintext

---

Getting Help

If encryption still fails:

1. Gather diagnostics:

   # Controller version
   kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
   
   # Certificate validity
   kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
   
   # Plugin version (in Headlamp UI)
   Settings → Sealed Secrets → About
   

2. Enable debug logging (see above)

3. Try kubeseal CLI to isolate issue

4. Report issue:

   • Plugin Issues
   • Controller Issues

Include:

• Browser and version
• Plugin version
• Controller version
• Full error message
• Browser console logs
• Steps to reproduce

See Also

• Common Errors - General troubleshooting
• Controller Issues - Controller-specific problems
• Encryption Flow - How encryption works
• Creating Secrets - Usage guide