docs: implement Phase 4 - troubleshooting guides and ADRs
Created comprehensive troubleshooting documentation: - docs/troubleshooting/README.md - Main troubleshooting hub - docs/troubleshooting/common-errors.md - Frequent errors and fixes - docs/troubleshooting/controller-issues.md - Controller problems - docs/troubleshooting/encryption-failures.md - Encryption debugging - docs/troubleshooting/permission-errors.md - RBAC troubleshooting Created Architecture Decision Records: - docs/architecture/adr/README.md - ADR index - docs/architecture/adr/001-result-types.md - Result<T,E> pattern - docs/architecture/adr/002-branded-types.md - Compile-time type safety - docs/architecture/adr/003-client-side-crypto.md - Browser encryption - docs/architecture/adr/004-rbac-integration.md - Permission-aware UI - docs/architecture/adr/005-react-hooks-extraction.md - Custom hooks Total: 11 files, 2,847 lines added Troubleshooting guides cover: - Plugin installation/loading issues - Controller deployment/connectivity problems - Encryption/certificate errors - RBAC permission diagnosis and fixes - Browser-specific issues - Network troubleshooting - Diagnostic commands and tools ADRs document key architectural decisions: - Why Result types for error handling (vs exceptions) - Why branded types for type safety (vs classes) - Why client-side encryption (vs server-side) - Why RBAC-aware UI (vs showing all actions) - Why custom React hooks (vs inline logic) Each ADR includes: - Context and problem statement - Decision and implementation - Consequences (positive/negative) - Alternatives considered with rationale - Real-world impact and examples Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
This commit is contained in:
@@ -0,0 +1,157 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
Common issues and solutions for the Headlamp Sealed Secrets plugin.
|
||||
|
||||
## Quick Links
|
||||
|
||||
- **[Common Errors](common-errors.md)** - Frequent error messages and fixes
|
||||
- **[Controller Issues](controller-issues.md)** - Connection and deployment problems
|
||||
- **[Encryption Failures](encryption-failures.md)** - Debugging encryption errors
|
||||
- **[Permission Errors](permission-errors.md)** - RBAC troubleshooting
|
||||
|
||||
## Quick Diagnosis
|
||||
|
||||
### Plugin Not Visible in Headlamp
|
||||
|
||||
**Symptoms**: "Sealed Secrets" not showing in sidebar
|
||||
|
||||
**Quick Checks**:
|
||||
```bash
|
||||
# 1. Check plugin directory exists
|
||||
ls -la ~/Library/Application\ Support/Headlamp/plugins/headlamp-sealed-secrets/
|
||||
|
||||
# 2. Check plugin files are present
|
||||
ls ~/Library/Application\ Support/Headlamp/plugins/headlamp-sealed-secrets/dist/
|
||||
|
||||
# 3. Check Headlamp version
|
||||
headlamp --version # Should be v0.13.0+
|
||||
```
|
||||
|
||||
**Solution**: See [Installation Guide](../getting-started/installation.md)
|
||||
|
||||
---
|
||||
|
||||
### Controller Not Found
|
||||
|
||||
**Symptoms**: "Failed to fetch controller certificate" or health status shows unhealthy
|
||||
|
||||
**Quick Checks**:
|
||||
```bash
|
||||
# Check controller is running
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# Check service exists
|
||||
kubectl get svc -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
**Solution**: See [Controller Issues](controller-issues.md)
|
||||
|
||||
---
|
||||
|
||||
### Permission Denied
|
||||
|
||||
**Symptoms**: "Forbidden" errors, missing buttons in UI
|
||||
|
||||
**Quick Checks**:
|
||||
```bash
|
||||
# Test your permissions
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com
|
||||
kubectl auth can-i get secrets
|
||||
```
|
||||
|
||||
**Solution**: See [Permission Errors](permission-errors.md)
|
||||
|
||||
---
|
||||
|
||||
### Encryption Fails
|
||||
|
||||
**Symptoms**: "Encryption failed" when creating sealed secrets
|
||||
|
||||
**Quick Checks**:
|
||||
```bash
|
||||
# Check certificate is valid
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates
|
||||
```
|
||||
|
||||
**Solution**: See [Encryption Failures](encryption-failures.md)
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If you can't find a solution:
|
||||
|
||||
1. **Check the logs**:
|
||||
```bash
|
||||
# Headlamp logs (depends on installation method)
|
||||
# For desktop app:
|
||||
tail -f ~/Library/Logs/Headlamp/main.log
|
||||
|
||||
# Controller logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
|
||||
2. **Enable browser console**:
|
||||
- View → Toggle Developer Tools
|
||||
- Look for errors in Console tab
|
||||
|
||||
3. **Search GitHub Issues**:
|
||||
- [Open Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)
|
||||
- [Closed Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues?q=is%3Aissue+is%3Aclosed)
|
||||
|
||||
4. **Ask for Help**:
|
||||
- [GitHub Discussions](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/discussions)
|
||||
- [Create New Issue](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues/new)
|
||||
|
||||
## Reporting Bugs
|
||||
|
||||
When reporting an issue, include:
|
||||
|
||||
- **Plugin version**: Check Settings page or `package.json`
|
||||
- **Headlamp version**: `headlamp --version`
|
||||
- **Kubernetes version**: `kubectl version --short`
|
||||
- **Controller version**: `kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'`
|
||||
- **Error messages**: Full error text from UI or console
|
||||
- **Browser console logs**: Copy from Developer Tools
|
||||
- **Steps to reproduce**: What you did before the error
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Error Message Format
|
||||
|
||||
Plugin errors typically follow this format:
|
||||
|
||||
```
|
||||
[Context]: Specific error message
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `Failed to fetch certificate: Network error`
|
||||
- `Validation failed: Name must be a valid DNS-1123 subdomain`
|
||||
- `Encryption failed: Invalid public key`
|
||||
|
||||
### Health Check Failures
|
||||
|
||||
The plugin checks controller health every 30 seconds. If health checks fail:
|
||||
|
||||
1. **Transient failures**: Wait 1-2 minutes for retry
|
||||
2. **Persistent failures**: Check controller status
|
||||
3. **Network issues**: Verify cluster connectivity
|
||||
|
||||
### RBAC Failures
|
||||
|
||||
Missing permissions hide UI elements:
|
||||
|
||||
| Permission Missing | UI Impact |
|
||||
|-------------------|-----------|
|
||||
| `list sealedsecrets` | No sealed secrets shown |
|
||||
| `create sealedsecrets` | "Create" button hidden |
|
||||
| `delete sealedsecrets` | "Delete" button disabled |
|
||||
| `get secrets` | "Decrypt" button hidden |
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[Common Errors](common-errors.md)** - Start with most frequent issues
|
||||
- **[User Guide](../user-guide/)** - Review feature documentation
|
||||
- **[GitHub Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)** - Search existing issues
|
||||
@@ -0,0 +1,561 @@
|
||||
# Common Errors
|
||||
|
||||
Frequent error messages and their solutions.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Plugin Errors](#plugin-errors)
|
||||
- [Controller Errors](#controller-errors)
|
||||
- [Encryption Errors](#encryption-errors)
|
||||
- [Permission Errors](#permission-errors)
|
||||
- [Validation Errors](#validation-errors)
|
||||
|
||||
---
|
||||
|
||||
## Plugin Errors
|
||||
|
||||
### "Plugin failed to load"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Error loading plugin headlamp-sealed-secrets: Invalid plugin manifest
|
||||
```
|
||||
|
||||
**Cause**: Corrupted or incompatible plugin installation
|
||||
|
||||
**Solution**:
|
||||
1. Remove the plugin:
|
||||
```bash
|
||||
rm -rf ~/Library/Application\ Support/Headlamp/plugins/headlamp-sealed-secrets/
|
||||
```
|
||||
|
||||
2. Reinstall from latest release:
|
||||
```bash
|
||||
curl -LO https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/releases/download/v0.2.0/headlamp-sealed-secrets-0.2.0.tar.gz
|
||||
tar -xzf headlamp-sealed-secrets-0.2.0.tar.gz -C ~/Library/Application\ Support/Headlamp/plugins/
|
||||
```
|
||||
|
||||
3. Restart Headlamp
|
||||
|
||||
---
|
||||
|
||||
### "Headlamp version incompatible"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Plugin requires Headlamp v0.13.0 or later (current: v0.12.0)
|
||||
```
|
||||
|
||||
**Cause**: Headlamp version too old
|
||||
|
||||
**Solution**:
|
||||
Upgrade Headlamp:
|
||||
```bash
|
||||
# macOS with Homebrew
|
||||
brew upgrade headlamp
|
||||
|
||||
# Or download from https://headlamp.dev/docs/latest/installation/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Controller Errors
|
||||
|
||||
### "Failed to fetch controller certificate"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Failed to fetch certificate: Service 'sealed-secrets-controller' not found in namespace 'kube-system'
|
||||
```
|
||||
|
||||
**Cause**: Sealed Secrets controller not installed
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Install controller
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
|
||||
|
||||
# Wait for controller to be ready
|
||||
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s
|
||||
|
||||
# Verify
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Controller health check failed"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Health check failed: Connection timeout after 3 attempts
|
||||
```
|
||||
|
||||
**Cause**: Controller not responding or network issues
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# 1. Check controller is running
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# 2. Check logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=50
|
||||
|
||||
# 3. Test direct connection
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
|
||||
# In another terminal:
|
||||
curl http://localhost:8080/v1/cert.pem
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
**If pod is not running**:
|
||||
```bash
|
||||
kubectl describe pod -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
Look for image pull errors, resource constraints, or CrashLoopBackOff.
|
||||
|
||||
**If pod is running but not responding**:
|
||||
```bash
|
||||
# Restart the controller
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Controller version mismatch"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Warning: Controller version v0.18.0 detected. Plugin tested with v0.24.0+
|
||||
```
|
||||
|
||||
**Cause**: Old controller version
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Upgrade controller (preserves existing secrets)
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
|
||||
|
||||
# Verify upgrade
|
||||
kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
|
||||
```
|
||||
|
||||
**Warning**: Backup sealing keys before upgrading:
|
||||
```bash
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Encryption Errors
|
||||
|
||||
### "Encryption failed: Invalid public key"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Encryption failed: Invalid public key format
|
||||
```
|
||||
|
||||
**Cause**: Corrupted or malformed certificate
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Fetch and validate certificate
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d > cert.pem
|
||||
openssl x509 -in cert.pem -noout -text
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
If certificate is invalid, the controller may be corrupted. Restart it:
|
||||
```bash
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Encryption failed: Certificate expired"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Encryption failed: Certificate expired on 2025-01-15
|
||||
```
|
||||
|
||||
**Cause**: Sealing key has expired (typically after 30 days of inactivity)
|
||||
|
||||
**Solution**:
|
||||
|
||||
**Option 1: Use existing valid certificate** (if you have multiple keys):
|
||||
```bash
|
||||
# List all certificates
|
||||
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
|
||||
|
||||
# Plugin will automatically use the newest valid certificate
|
||||
```
|
||||
|
||||
**Option 2: Rotate sealing keys**:
|
||||
```bash
|
||||
# Generate new key (requires cluster-admin)
|
||||
kubectl delete secret -n kube-system sealed-secrets-key
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Wait for new key generation
|
||||
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s
|
||||
```
|
||||
|
||||
**Warning**: After key rotation, existing SealedSecrets remain valid but cannot be modified. See [Secret Rotation Tutorial](../tutorials/secret-rotation.md).
|
||||
|
||||
---
|
||||
|
||||
### "Value too large"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Encryption failed: Value exceeds maximum size (1MB)
|
||||
```
|
||||
|
||||
**Cause**: Secret value larger than 1MB (RSA encryption limit)
|
||||
|
||||
**Solution**:
|
||||
|
||||
**For large files**: Store in external secret management (Vault, AWS Secrets Manager) and reference:
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: large-secret-ref
|
||||
stringData:
|
||||
vault-path: "secret/data/myapp/config"
|
||||
```
|
||||
|
||||
**For multiple keys**: Split into separate secrets:
|
||||
```bash
|
||||
# Instead of one large secret:
|
||||
# - config.json (500KB)
|
||||
# - data.bin (600KB)
|
||||
|
||||
# Create two secrets:
|
||||
# - myapp-config (config.json)
|
||||
# - myapp-data (data.bin)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Permission Errors
|
||||
|
||||
### "Forbidden: User cannot list SealedSecrets"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Failed to load sealed secrets: Forbidden (403)
|
||||
User 'alice' cannot list resource 'sealedsecrets' in API group 'bitnami.com'
|
||||
```
|
||||
|
||||
**Cause**: Missing RBAC permissions
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check your permissions
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com --all-namespaces
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
|
||||
Apply this ClusterRole (requires cluster-admin):
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-viewer
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-sealed-secrets-viewer
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-viewer
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Apply:
|
||||
```bash
|
||||
kubectl apply -f sealed-secrets-viewer-rbac.yaml
|
||||
```
|
||||
|
||||
See [RBAC Permissions Guide](../user-guide/rbac-permissions.md) for detailed examples.
|
||||
|
||||
---
|
||||
|
||||
### "Cannot download certificate: Service access denied"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Failed to fetch certificate: Forbidden (403)
|
||||
User cannot access service/proxy 'sealed-secrets-controller'
|
||||
```
|
||||
|
||||
**Cause**: Missing service access permission
|
||||
|
||||
**Solution**:
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-cert-downloader
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-cert-downloader
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-cert-downloader
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Errors
|
||||
|
||||
### "Invalid name format"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Validation failed: Name must be a valid DNS-1123 subdomain (lowercase alphanumeric, '-', '.', max 253 chars)
|
||||
```
|
||||
|
||||
**Cause**: Secret name doesn't follow Kubernetes naming rules
|
||||
|
||||
**Invalid Names**:
|
||||
- `My_Secret` (uppercase, underscore)
|
||||
- `secret@prod` (special characters)
|
||||
- `Secret-Name` (uppercase)
|
||||
- `-secret` (starts with hyphen)
|
||||
|
||||
**Valid Names**:
|
||||
- `my-secret`
|
||||
- `prod.secret`
|
||||
- `secret-123`
|
||||
- `my-app-secret`
|
||||
|
||||
**Rules**:
|
||||
- Only lowercase letters, numbers, hyphens, dots
|
||||
- Start and end with alphanumeric
|
||||
- Max 253 characters
|
||||
- Must match: `[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*`
|
||||
|
||||
---
|
||||
|
||||
### "Empty secret value"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Validation failed: Secret value cannot be empty
|
||||
```
|
||||
|
||||
**Cause**: Trying to create a secret with empty value
|
||||
|
||||
**Solution**:
|
||||
Either provide a value or use a placeholder:
|
||||
```yaml
|
||||
# If you need a placeholder for later update:
|
||||
stringData:
|
||||
password: "changeme"
|
||||
```
|
||||
|
||||
Note: Kubernetes allows empty values, but it's usually not intentional.
|
||||
|
||||
---
|
||||
|
||||
### "Invalid namespace"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Validation failed: Namespace does not exist or is invalid
|
||||
```
|
||||
|
||||
**Cause**: Target namespace doesn't exist
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# List all namespaces
|
||||
kubectl get namespaces
|
||||
|
||||
# Check specific namespace
|
||||
kubectl get namespace production
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
Create the namespace first:
|
||||
```bash
|
||||
kubectl create namespace production
|
||||
```
|
||||
|
||||
Or use an existing namespace:
|
||||
```bash
|
||||
kubectl get namespaces
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Browser-Specific Errors
|
||||
|
||||
### "localStorage not available"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Failed to save settings: localStorage is not available
|
||||
```
|
||||
|
||||
**Cause**: Browser privacy settings blocking localStorage
|
||||
|
||||
**Solution**:
|
||||
|
||||
**Safari**:
|
||||
1. Preferences → Privacy
|
||||
2. Uncheck "Prevent cross-site tracking"
|
||||
3. Reload Headlamp
|
||||
|
||||
**Chrome**:
|
||||
1. Settings → Privacy and security
|
||||
2. Cookies and other site data
|
||||
3. Select "Allow all cookies"
|
||||
4. Reload Headlamp
|
||||
|
||||
**Firefox**:
|
||||
1. Preferences → Privacy & Security
|
||||
2. Enhanced Tracking Protection → Standard
|
||||
3. Reload Headlamp
|
||||
|
||||
---
|
||||
|
||||
### "Certificate validation failed"
|
||||
|
||||
**Full Error** (Browser Console):
|
||||
```
|
||||
Certificate validation failed: unable to verify the first certificate
|
||||
```
|
||||
|
||||
**Cause**: Self-signed Kubernetes API certificate
|
||||
|
||||
**Solution**:
|
||||
|
||||
This is expected with many Kubernetes clusters. The plugin handles this internally.
|
||||
|
||||
If you see this in browser console but plugin works, you can ignore it.
|
||||
|
||||
If plugin doesn't work:
|
||||
1. Use `kubectl proxy` for local development
|
||||
2. Configure Headlamp to trust cluster certificate
|
||||
|
||||
---
|
||||
|
||||
## Network Errors
|
||||
|
||||
### "Connection timeout"
|
||||
|
||||
**Full Error**:
|
||||
```
|
||||
Failed to fetch certificate: Connection timeout after 30000ms
|
||||
```
|
||||
|
||||
**Cause**: Network connectivity issues
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Test cluster connectivity
|
||||
kubectl cluster-info
|
||||
|
||||
# Test service connectivity
|
||||
kubectl get svc -n kube-system sealed-secrets-controller
|
||||
|
||||
# Port-forward and test manually
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
|
||||
curl http://localhost:8080/v1/cert.pem
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
**VPN Issues**: If using VPN, ensure it's connected
|
||||
**Proxy**: Configure proxy in Headlamp settings
|
||||
**Firewall**: Check firewall allows Kubernetes API access
|
||||
|
||||
---
|
||||
|
||||
### "CORS error"
|
||||
|
||||
**Full Error** (Browser Console):
|
||||
```
|
||||
Access to fetch at 'https://kubernetes.default.svc' from origin 'http://localhost:3000' has been blocked by CORS policy
|
||||
```
|
||||
|
||||
**Cause**: Browser security blocking cross-origin requests
|
||||
|
||||
**Solution**:
|
||||
|
||||
This should not happen in production. If you see this during development:
|
||||
|
||||
1. Use Headlamp's built-in proxy (recommended)
|
||||
2. Or configure Kubernetes API server with CORS headers (not recommended)
|
||||
|
||||
---
|
||||
|
||||
## Getting More Help
|
||||
|
||||
If your error isn't listed:
|
||||
|
||||
1. **Search GitHub Issues**: [https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)
|
||||
|
||||
2. **Check Controller Logs**:
|
||||
```bash
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=100
|
||||
```
|
||||
|
||||
3. **Enable Debug Logging** (browser console):
|
||||
```javascript
|
||||
localStorage.setItem('debug', 'headlamp-sealed-secrets:*')
|
||||
```
|
||||
Then reload Headlamp.
|
||||
|
||||
4. **Create New Issue**: [https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues/new](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues/new)
|
||||
|
||||
Include:
|
||||
- Full error message
|
||||
- Plugin version
|
||||
- Headlamp version
|
||||
- Kubernetes version
|
||||
- Controller version
|
||||
- Steps to reproduce
|
||||
|
||||
## See Also
|
||||
|
||||
- [Controller Issues](controller-issues.md) - Controller-specific problems
|
||||
- [Encryption Failures](encryption-failures.md) - Debugging encryption
|
||||
- [Permission Errors](permission-errors.md) - RBAC troubleshooting
|
||||
- [User Guide](../user-guide/) - Feature documentation
|
||||
@@ -0,0 +1,585 @@
|
||||
# Controller Issues
|
||||
|
||||
Troubleshooting Sealed Secrets controller problems.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Controller Not Found](#controller-not-found)
|
||||
- [Controller Not Starting](#controller-not-starting)
|
||||
- [Controller Crashing](#controller-crashing)
|
||||
- [Certificate Issues](#certificate-issues)
|
||||
- [Performance Issues](#performance-issues)
|
||||
- [Upgrade Issues](#upgrade-issues)
|
||||
|
||||
---
|
||||
|
||||
## Controller Not Found
|
||||
|
||||
### Symptom
|
||||
|
||||
Plugin shows "Controller not found" or health status is unhealthy.
|
||||
|
||||
### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check if controller exists
|
||||
kubectl get deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Check service
|
||||
kubectl get svc -n kube-system sealed-secrets-controller
|
||||
|
||||
# Check pods
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
|
||||
### Solutions
|
||||
|
||||
#### Controller Not Installed
|
||||
|
||||
Install the controller:
|
||||
|
||||
```bash
|
||||
# Install latest version
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
|
||||
|
||||
# Wait for deployment
|
||||
kubectl wait --for=condition=available deployment/sealed-secrets-controller -n kube-system --timeout=60s
|
||||
|
||||
# Verify
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
|
||||
#### Wrong Namespace
|
||||
|
||||
If controller is in different namespace:
|
||||
|
||||
```bash
|
||||
# Find controller
|
||||
kubectl get deployments --all-namespaces -l name=sealed-secrets-controller
|
||||
|
||||
# Update plugin settings (in Headlamp UI):
|
||||
# Settings → Sealed Secrets → Controller Namespace: <your-namespace>
|
||||
```
|
||||
|
||||
#### Custom Service Name
|
||||
|
||||
If service has custom name:
|
||||
|
||||
```bash
|
||||
# Find service
|
||||
kubectl get svc --all-namespaces -l name=sealed-secrets-controller
|
||||
|
||||
# Update plugin settings (in Headlamp UI):
|
||||
# Settings → Sealed Secrets → Service Name: <your-service-name>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Controller Not Starting
|
||||
|
||||
### Symptom
|
||||
|
||||
Controller pod shows `Pending`, `ContainerCreating`, or `ImagePullBackOff`.
|
||||
|
||||
### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check pod status
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# Get detailed status
|
||||
kubectl describe pod -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# Check events
|
||||
kubectl get events -n kube-system --sort-by='.lastTimestamp' | grep sealed-secrets
|
||||
```
|
||||
|
||||
### Common Causes
|
||||
|
||||
#### ImagePullBackOff
|
||||
|
||||
**Cause**: Cannot pull container image
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl describe pod -n kube-system -l name=sealed-secrets-controller | grep -A 5 "Events:"
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
**Private registry authentication**:
|
||||
```bash
|
||||
# Create image pull secret
|
||||
kubectl create secret docker-registry regcred \
|
||||
--docker-server=<registry> \
|
||||
--docker-username=<username> \
|
||||
--docker-password=<password> \
|
||||
-n kube-system
|
||||
|
||||
# Update deployment
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'
|
||||
```
|
||||
|
||||
**Network issues**: Check cluster can reach `quay.io` or your registry.
|
||||
|
||||
**Wrong image tag**: Verify image exists:
|
||||
```bash
|
||||
kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
|
||||
```
|
||||
|
||||
#### Insufficient Resources
|
||||
|
||||
**Cause**: Node doesn't have enough CPU/memory
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl describe pod -n kube-system -l name=sealed-secrets-controller | grep -A 5 "FailedScheduling"
|
||||
```
|
||||
|
||||
**Solution**: Lower resource requests or add nodes:
|
||||
```bash
|
||||
# Lower requests (not recommended for production)
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '
|
||||
{
|
||||
"spec": {
|
||||
"template": {
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "sealed-secrets-controller",
|
||||
"resources": {
|
||||
"requests": {
|
||||
"cpu": "50m",
|
||||
"memory": "64Mi"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
#### PVC Issues
|
||||
|
||||
**Cause**: PersistentVolumeClaim not bound (if using custom storage)
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl get pvc -n kube-system
|
||||
```
|
||||
|
||||
**Solution**: Ensure StorageClass exists and volumes are available.
|
||||
|
||||
---
|
||||
|
||||
## Controller Crashing
|
||||
|
||||
### Symptom
|
||||
|
||||
Controller pod shows `CrashLoopBackOff` or restarts frequently.
|
||||
|
||||
### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check restart count
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# View recent logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=100
|
||||
|
||||
# View previous crash logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --previous
|
||||
```
|
||||
|
||||
### Common Causes
|
||||
|
||||
#### Invalid Sealing Key
|
||||
|
||||
**Error in logs**:
|
||||
```
|
||||
Error loading sealed secrets key: invalid PEM data
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Backup existing key (if valid)
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o yaml > backup.yaml
|
||||
|
||||
# Delete corrupted key
|
||||
kubectl delete secret -n kube-system sealed-secrets-key
|
||||
|
||||
# Restart controller to generate new key
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Wait for new key
|
||||
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s
|
||||
```
|
||||
|
||||
**Warning**: This generates a new key. Existing SealedSecrets will still work but cannot be modified.
|
||||
|
||||
#### Certificate Conflict
|
||||
|
||||
**Error in logs**:
|
||||
```
|
||||
Multiple certificates found, unable to determine active key
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# List all sealing keys
|
||||
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
|
||||
|
||||
# Remove old keys (keep backup!)
|
||||
kubectl delete secret -n kube-system <old-key-name>
|
||||
```
|
||||
|
||||
#### Memory Issues
|
||||
|
||||
**Error in logs**:
|
||||
```
|
||||
OOMKilled
|
||||
```
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl describe pod -n kube-system -l name=sealed-secrets-controller | grep -A 5 "Last State"
|
||||
```
|
||||
|
||||
**Solution**: Increase memory limits:
|
||||
```bash
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '
|
||||
{
|
||||
"spec": {
|
||||
"template": {
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "sealed-secrets-controller",
|
||||
"resources": {
|
||||
"limits": {
|
||||
"memory": "512Mi"
|
||||
},
|
||||
"requests": {
|
||||
"memory": "256Mi"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
#### RBAC Issues
|
||||
|
||||
**Error in logs**:
|
||||
```
|
||||
Forbidden: cannot list resource "secrets" in API group ""
|
||||
```
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl get clusterrole sealed-secrets-controller
|
||||
kubectl get clusterrolebinding sealed-secrets-controller
|
||||
```
|
||||
|
||||
**Solution**: Reapply controller manifest:
|
||||
```bash
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Certificate Issues
|
||||
|
||||
### Expired Certificate
|
||||
|
||||
**Symptom**: "Certificate expired" error when encrypting
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# Get certificate expiry
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | \
|
||||
base64 -d | \
|
||||
openssl x509 -noout -enddate
|
||||
```
|
||||
|
||||
**Solution**: Rotate keys (see [Secret Rotation Tutorial](../tutorials/secret-rotation.md))
|
||||
|
||||
```bash
|
||||
# Generate new key (keeps old for decryption)
|
||||
kubectl annotate secret -n kube-system sealed-secrets-key \
|
||||
sealedsecrets.bitnami.com/sealed-secrets-key-rotation=rotate
|
||||
|
||||
# Or delete and recreate
|
||||
kubectl delete secret -n kube-system sealed-secrets-key
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
### Multiple Certificates
|
||||
|
||||
**Symptom**: Inconsistent encryption results
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# List all certificates
|
||||
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
|
||||
|
||||
# View details
|
||||
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml
|
||||
```
|
||||
|
||||
**Solution**: Controller uses newest valid certificate. This is normal after key rotation.
|
||||
|
||||
To clean up old keys (after backup):
|
||||
```bash
|
||||
# Keep newest 2 keys, delete older ones
|
||||
kubectl delete secret -n kube-system <old-key-name>
|
||||
```
|
||||
|
||||
### Certificate Not Found
|
||||
|
||||
**Symptom**: "No valid certificate found"
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
kubectl get secret -n kube-system sealed-secrets-key
|
||||
```
|
||||
|
||||
**Solution**: Restart controller to generate:
|
||||
```bash
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Slow Secret Unsealing
|
||||
|
||||
**Symptom**: SealedSecrets take minutes to unseal
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check controller CPU/memory usage
|
||||
kubectl top pod -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# Check events
|
||||
kubectl get events -n kube-system --sort-by='.lastTimestamp' | grep sealed-secrets
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
#### Increase Resources
|
||||
|
||||
```bash
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '
|
||||
{
|
||||
"spec": {
|
||||
"template": {
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "sealed-secrets-controller",
|
||||
"resources": {
|
||||
"limits": {
|
||||
"cpu": "1000m",
|
||||
"memory": "512Mi"
|
||||
},
|
||||
"requests": {
|
||||
"cpu": "500m",
|
||||
"memory": "256Mi"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
#### Check Pod Placement
|
||||
|
||||
```bash
|
||||
# Get node
|
||||
kubectl get pod -n kube-system -l name=sealed-secrets-controller -o wide
|
||||
|
||||
# Check node load
|
||||
kubectl top node <node-name>
|
||||
```
|
||||
|
||||
Consider node affinity if node is overloaded.
|
||||
|
||||
### Health Checks Timing Out
|
||||
|
||||
**Symptom**: Plugin shows controller as unhealthy despite it working
|
||||
|
||||
**Cause**: Slow network or overloaded controller
|
||||
|
||||
**Solution**: Increase health check timeout in plugin settings (Headlamp UI):
|
||||
- Settings → Sealed Secrets → Health Check Timeout: 60s (default: 30s)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Issues
|
||||
|
||||
### Upgrade Failed
|
||||
|
||||
**Symptom**: Controller won't upgrade or crashes after upgrade
|
||||
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check deployment history
|
||||
kubectl rollout history deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Check current image
|
||||
kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
|
||||
```
|
||||
|
||||
**Solution**: Rollback and retry:
|
||||
```bash
|
||||
# Rollback to previous version
|
||||
kubectl rollout undo deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Wait for rollback
|
||||
kubectl rollout status deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller
|
||||
```
|
||||
|
||||
### Version Compatibility
|
||||
|
||||
**Check compatibility**:
|
||||
|
||||
| Plugin Version | Controller Version | Compatible |
|
||||
|---------------|-------------------|------------|
|
||||
| v0.2.0 | v0.24.0+ | ✅ Yes |
|
||||
| v0.2.0 | v0.20.0 - v0.23.x | ⚠️ Limited |
|
||||
| v0.2.0 | < v0.20.0 | ❌ No |
|
||||
|
||||
**Upgrade controller**:
|
||||
```bash
|
||||
# Backup sealing keys first!
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-backup.yaml
|
||||
|
||||
# Upgrade
|
||||
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
|
||||
|
||||
# Verify
|
||||
kubectl rollout status deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
### Lost Sealing Keys After Upgrade
|
||||
|
||||
**Symptom**: Existing SealedSecrets won't unseal after upgrade
|
||||
|
||||
**Cause**: Sealing keys deleted during upgrade
|
||||
|
||||
**Recovery**:
|
||||
|
||||
If you have backup:
|
||||
```bash
|
||||
# Restore keys
|
||||
kubectl apply -f sealed-secrets-backup.yaml
|
||||
|
||||
# Restart controller
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
If no backup, keys are **permanently lost**. You must:
|
||||
1. Delete all SealedSecrets
|
||||
2. Recreate them with new keys
|
||||
3. See [Disaster Recovery Tutorial](../tutorials/disaster-recovery.md)
|
||||
|
||||
---
|
||||
|
||||
## Debugging Tools
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
```bash
|
||||
# Add debug flag to controller
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '
|
||||
{
|
||||
"spec": {
|
||||
"template": {
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "sealed-secrets-controller",
|
||||
"args": ["--update-status", "--v=5"]
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
|
||||
# View debug logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller -f
|
||||
```
|
||||
|
||||
### Port-Forward for Testing
|
||||
|
||||
```bash
|
||||
# Forward controller port locally
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
|
||||
|
||||
# Test certificate endpoint
|
||||
curl http://localhost:8080/v1/cert.pem
|
||||
|
||||
# Test health
|
||||
curl http://localhost:8080/healthz
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
If Prometheus is installed:
|
||||
|
||||
```bash
|
||||
# Enable metrics
|
||||
kubectl patch deployment sealed-secrets-controller -n kube-system -p '
|
||||
{
|
||||
"spec": {
|
||||
"template": {
|
||||
"spec": {
|
||||
"containers": [{
|
||||
"name": "sealed-secrets-controller",
|
||||
"args": ["--update-status", "--metrics-addr=:8081"]
|
||||
}]
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
|
||||
# Access metrics
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8081:8081
|
||||
curl http://localhost:8081/metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If issues persist:
|
||||
|
||||
1. **Gather diagnostic info**:
|
||||
```bash
|
||||
# Create diagnostic bundle
|
||||
kubectl get all -n kube-system -l name=sealed-secrets-controller -o yaml > controller-diagnostics.yaml
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=500 > controller-logs.txt
|
||||
kubectl describe deployment -n kube-system sealed-secrets-controller > controller-describe.txt
|
||||
```
|
||||
|
||||
2. **Check Sealed Secrets project**:
|
||||
- [GitHub Issues](https://github.com/bitnami-labs/sealed-secrets/issues)
|
||||
- [Documentation](https://github.com/bitnami-labs/sealed-secrets#sealed-secrets)
|
||||
|
||||
3. **Plugin-specific issues**:
|
||||
- [Plugin Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)
|
||||
- Include diagnostic bundle when reporting
|
||||
|
||||
## See Also
|
||||
|
||||
- [Common Errors](common-errors.md) - General error messages
|
||||
- [Encryption Failures](encryption-failures.md) - Encryption-specific issues
|
||||
- [Disaster Recovery](../tutorials/disaster-recovery.md) - Backup and restore
|
||||
- [Secret Rotation](../tutorials/secret-rotation.md) - Key rotation procedures
|
||||
@@ -0,0 +1,654 @@
|
||||
# Encryption Failures
|
||||
|
||||
Debugging encryption errors when creating sealed secrets.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Understanding the Encryption Flow](#understanding-the-encryption-flow)
|
||||
- [Certificate Problems](#certificate-problems)
|
||||
- [Value Issues](#value-issues)
|
||||
- [Scope Problems](#scope-problems)
|
||||
- [Browser Issues](#browser-issues)
|
||||
- [Network Issues](#network-issues)
|
||||
|
||||
---
|
||||
|
||||
## Understanding the Encryption Flow
|
||||
|
||||
Before troubleshooting, understand how encryption works:
|
||||
|
||||
```
|
||||
1. Plugin fetches public certificate from controller
|
||||
GET /api/v1/namespaces/kube-system/services/sealed-secrets-controller:http/proxy/v1/cert.pem
|
||||
|
||||
2. Plugin validates certificate (PEM format, expiry, fingerprint)
|
||||
|
||||
3. Plugin encrypts value client-side using RSA-OAEP
|
||||
- Generates random AES-256 key
|
||||
- Encrypts value with AES-256-GCM
|
||||
- Encrypts AES key with RSA public key
|
||||
- Combines into encrypted payload
|
||||
|
||||
4. Plugin creates SealedSecret with encrypted data
|
||||
POST /apis/bitnami.com/v1alpha1/namespaces/<ns>/sealedsecrets
|
||||
|
||||
5. Controller unseals secret server-side
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- Encryption happens **in browser** (plaintext never leaves your machine)
|
||||
- Certificate must be **valid PEM format** and **not expired**
|
||||
- Maximum value size: **~1MB** (RSA limitation)
|
||||
|
||||
---
|
||||
|
||||
## Certificate Problems
|
||||
|
||||
### "Failed to fetch certificate"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Failed to fetch certificate: Network error
|
||||
```
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# 1. Check controller is running
|
||||
kubectl get pods -n kube-system -l name=sealed-secrets-controller
|
||||
|
||||
# 2. Test certificate endpoint directly
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
|
||||
# In another terminal:
|
||||
curl http://localhost:8080/v1/cert.pem
|
||||
```
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Controller not running**: See [Controller Issues](controller-issues.md)
|
||||
|
||||
**Certificate endpoint not responding**:
|
||||
```bash
|
||||
# Check controller logs
|
||||
kubectl logs -n kube-system -l name=sealed-secrets-controller --tail=50
|
||||
|
||||
# Restart controller
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
**RBAC permission denied**:
|
||||
```bash
|
||||
# Check service access permission
|
||||
kubectl auth can-i get services/sealed-secrets-controller -n kube-system
|
||||
|
||||
# If no, apply RBAC (requires cluster-admin):
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-cert-downloader
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services", "services/proxy"]
|
||||
verbs: ["get"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: $(kubectl config view --minify -o jsonpath='{.contexts[0].context.user}')-cert-downloader
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-cert-downloader
|
||||
subjects:
|
||||
- kind: User
|
||||
name: $(kubectl config view --minify -o jsonpath='{.contexts[0].context.user}')
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Certificate expired"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Certificate expired on 2025-01-15T10:30:00Z
|
||||
```
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check certificate expiry
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | \
|
||||
base64 -d | \
|
||||
openssl x509 -noout -dates
|
||||
|
||||
# Example output:
|
||||
# notBefore=Jan 1 00:00:00 2025 GMT
|
||||
# notAfter=Jan 15 10:30:00 2025 GMT ← Expired!
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
Rotate sealing keys (see [Secret Rotation Tutorial](../tutorials/secret-rotation.md)):
|
||||
|
||||
```bash
|
||||
# Option 1: Delete old key (generates new automatically)
|
||||
kubectl delete secret -n kube-system sealed-secrets-key
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Option 2: Annotate for rotation (keeps old for decryption)
|
||||
kubectl annotate secret -n kube-system sealed-secrets-key \
|
||||
sealedsecrets.bitnami.com/sealed-secrets-key-rotation=rotate
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
|
||||
# Wait for new key
|
||||
kubectl wait --for=condition=ready pod -n kube-system -l name=sealed-secrets-controller --timeout=60s
|
||||
|
||||
# Verify new certificate
|
||||
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
|
||||
```
|
||||
|
||||
**Warning**: After key rotation:
|
||||
- ✅ Existing SealedSecrets continue to work (controller keeps old key for decryption)
|
||||
- ❌ Cannot modify existing SealedSecrets (must delete and recreate with new key)
|
||||
|
||||
---
|
||||
|
||||
### "Invalid PEM format"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Certificate is not valid PEM format
|
||||
```
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Fetch and validate certificate
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d > cert.pem
|
||||
|
||||
# Should start with:
|
||||
# -----BEGIN CERTIFICATE-----
|
||||
# Should end with:
|
||||
# -----END CERTIFICATE-----
|
||||
|
||||
cat cert.pem
|
||||
```
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Corrupted certificate**:
|
||||
```bash
|
||||
# Regenerate certificate
|
||||
kubectl delete secret -n kube-system sealed-secrets-key
|
||||
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
||||
```
|
||||
|
||||
**Wrong secret**: Ensure you're using correct secret:
|
||||
```bash
|
||||
# List all sealing keys
|
||||
kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
|
||||
|
||||
# Should show sealed-secrets-key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Certificate expiring soon" Warning
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
⚠️ Warning: Certificate expires in 15 days
|
||||
```
|
||||
|
||||
This is an **advance warning**, not an error. Plugin warns 30 days before expiry.
|
||||
|
||||
#### Action
|
||||
|
||||
Plan key rotation before expiry:
|
||||
|
||||
1. **Schedule maintenance window**
|
||||
2. **Backup existing keys**:
|
||||
```bash
|
||||
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-keys-backup.yaml
|
||||
```
|
||||
3. **Rotate keys**: See [Secret Rotation Tutorial](../tutorials/secret-rotation.md)
|
||||
4. **Recreate SealedSecrets** if needed
|
||||
|
||||
---
|
||||
|
||||
## Value Issues
|
||||
|
||||
### "Value too large"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Value exceeds maximum size (1MB)
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
RSA encryption has size limit (~1MB for RSA-2048).
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Split into multiple secrets**:
|
||||
```yaml
|
||||
# Instead of:
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: large-config
|
||||
data:
|
||||
config.json: <2MB base64 data> ❌
|
||||
|
||||
# Use:
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: config-part1
|
||||
data:
|
||||
config-part1.json: <500KB base64 data> ✅
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: config-part2
|
||||
data:
|
||||
config-part2.json: <500KB base64 data> ✅
|
||||
```
|
||||
|
||||
**Use external secret management** for very large files:
|
||||
- HashiCorp Vault
|
||||
- AWS Secrets Manager
|
||||
- Azure Key Vault
|
||||
- GCP Secret Manager
|
||||
|
||||
Then store only references in SealedSecrets:
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: external-ref
|
||||
stringData:
|
||||
vault-path: "secret/data/myapp/large-config"
|
||||
```
|
||||
|
||||
**Compress data** before encrypting:
|
||||
```bash
|
||||
# Compress config file
|
||||
gzip -c config.json | base64
|
||||
|
||||
# Then use compressed value in plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Empty value"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Validation failed: Secret value cannot be empty
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Trying to create secret with empty value.
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Provide a value**:
|
||||
```yaml
|
||||
stringData:
|
||||
password: "mysecretvalue" ✅
|
||||
```
|
||||
|
||||
**Use placeholder** if value unknown:
|
||||
```yaml
|
||||
stringData:
|
||||
password: "changeme" # Replace later
|
||||
```
|
||||
|
||||
**Remove key** if not needed:
|
||||
Don't create keys with empty values.
|
||||
|
||||
---
|
||||
|
||||
### "Invalid characters"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Value contains invalid characters
|
||||
```
|
||||
|
||||
This error is **rare** (plugin accepts any binary data). If you see it:
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Base64-encode first** (for binary data):
|
||||
```bash
|
||||
# Encode binary file
|
||||
base64 < binary-file.bin
|
||||
|
||||
# Use in plugin with base64-encoded value
|
||||
```
|
||||
|
||||
**Check for null bytes** in text:
|
||||
```bash
|
||||
# Remove null bytes
|
||||
tr -d '\0' < file.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scope Problems
|
||||
|
||||
### "Scope validation failed"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Validation failed: Invalid scope for namespace-scoped secret
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Scope mismatch between plugin setting and cluster policy.
|
||||
|
||||
#### Understanding Scopes
|
||||
|
||||
| Scope | Can Rename? | Can Move Namespace? | Use Case |
|
||||
|-------|------------|---------------------|----------|
|
||||
| `strict` | ❌ No | ❌ No | Production secrets |
|
||||
| `namespace-wide` | ✅ Yes | ❌ No | Dev environments |
|
||||
| `cluster-wide` | ✅ Yes | ✅ Yes | Shared configs |
|
||||
|
||||
#### Solution
|
||||
|
||||
**Use strict scope** (most secure):
|
||||
```yaml
|
||||
# In plugin:
|
||||
Scope: strict
|
||||
Name: my-secret
|
||||
Namespace: production
|
||||
```
|
||||
|
||||
**Match cluster policy**: If cluster enforces specific scope, use that:
|
||||
```bash
|
||||
# Check cluster policy
|
||||
kubectl get sealedsecrets.bitnami.com -o yaml | grep -A 5 scope
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Scope annotation missing"
|
||||
|
||||
#### Symptom
|
||||
|
||||
SealedSecret created but controller doesn't unseal it.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check SealedSecret
|
||||
kubectl get sealedsecret my-secret -n default -o yaml
|
||||
|
||||
# Look for annotation:
|
||||
metadata:
|
||||
annotations:
|
||||
sealedsecrets.bitnami.com/scope: strict ← Should be present
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
Plugin automatically adds scope annotation. If missing:
|
||||
|
||||
1. **Delete SealedSecret**:
|
||||
```bash
|
||||
kubectl delete sealedsecret my-secret -n default
|
||||
```
|
||||
|
||||
2. **Recreate with plugin** (don't create manually)
|
||||
|
||||
---
|
||||
|
||||
## Browser Issues
|
||||
|
||||
### "Crypto API not available"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Web Crypto API not available
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Browser doesn't support Web Crypto API (very old browser).
|
||||
|
||||
#### Solution
|
||||
|
||||
**Upgrade browser**:
|
||||
- Chrome 37+
|
||||
- Firefox 34+
|
||||
- Safari 11+
|
||||
- Edge 79+
|
||||
|
||||
**Use HTTPS**: Web Crypto requires HTTPS (or localhost):
|
||||
```bash
|
||||
# If using custom Headlamp deployment, enable HTTPS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Encryption timeout"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Encryption failed: Operation timeout after 30000ms
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Browser crypto operation taking too long (rare).
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Reduce value size**: If encrypting large value, split it.
|
||||
|
||||
**Close other tabs**: Free up browser resources.
|
||||
|
||||
**Try different browser**: Some browsers have better crypto performance.
|
||||
|
||||
**Check browser console** for JavaScript errors:
|
||||
1. View → Toggle Developer Tools
|
||||
2. Console tab
|
||||
3. Look for errors during encryption
|
||||
|
||||
---
|
||||
|
||||
## Network Issues
|
||||
|
||||
### "Certificate fetch timeout"
|
||||
|
||||
#### Symptom
|
||||
```
|
||||
Failed to fetch certificate: Request timeout after 30000ms
|
||||
```
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Test cluster connectivity
|
||||
kubectl cluster-info
|
||||
|
||||
# Test service endpoint
|
||||
kubectl get svc -n kube-system sealed-secrets-controller
|
||||
|
||||
# Test with curl
|
||||
kubectl port-forward -n kube-system service/sealed-secrets-controller 8080:8080
|
||||
curl -m 5 http://localhost:8080/v1/cert.pem
|
||||
```
|
||||
|
||||
#### Solutions
|
||||
|
||||
**Slow network**: Increase timeout in plugin settings:
|
||||
- Settings → Sealed Secrets → Network Timeout: 60s
|
||||
|
||||
**VPN disconnected**: Reconnect VPN if using one.
|
||||
|
||||
**Firewall blocking**: Check firewall allows Kubernetes API access.
|
||||
|
||||
---
|
||||
|
||||
### "CORS error"
|
||||
|
||||
#### Symptom (Browser Console)
|
||||
```
|
||||
Access blocked by CORS policy
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Cross-origin request blocked (should not happen in normal usage).
|
||||
|
||||
#### Solution
|
||||
|
||||
**Use Headlamp's built-in proxy** (enabled by default).
|
||||
|
||||
If developing plugin locally:
|
||||
```bash
|
||||
# Start Headlamp in dev mode
|
||||
npm start
|
||||
|
||||
# Headlamp automatically proxies requests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging Encryption
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
In browser console:
|
||||
```javascript
|
||||
// Enable debug logs
|
||||
localStorage.setItem('debug', 'headlamp-sealed-secrets:*')
|
||||
|
||||
// Reload page
|
||||
location.reload()
|
||||
|
||||
// Try encryption again
|
||||
// Check console for detailed logs
|
||||
```
|
||||
|
||||
### Manual Encryption Test
|
||||
|
||||
Test encryption manually:
|
||||
|
||||
```javascript
|
||||
// In browser console
|
||||
const cert = await fetch('/api/v1/namespaces/kube-system/services/sealed-secrets-controller:http/proxy/v1/cert.pem')
|
||||
.then(r => r.text());
|
||||
|
||||
console.log('Certificate:', cert);
|
||||
|
||||
// Should show PEM certificate starting with:
|
||||
// -----BEGIN CERTIFICATE-----
|
||||
```
|
||||
|
||||
### Verify Encrypted Output
|
||||
|
||||
```bash
|
||||
# After creating SealedSecret, verify it
|
||||
kubectl get sealedsecret my-secret -n default -o yaml
|
||||
|
||||
# Should show:
|
||||
# spec:
|
||||
# encryptedData:
|
||||
# password: AgB... (base64 encrypted value)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Debugging
|
||||
|
||||
### Test with kubeseal CLI
|
||||
|
||||
If plugin fails but kubeseal works, it's a plugin issue:
|
||||
|
||||
```bash
|
||||
# Install kubeseal
|
||||
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
|
||||
tar -xzf kubeseal-0.24.0-linux-amd64.tar.gz
|
||||
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
|
||||
|
||||
# Test encryption
|
||||
echo -n mysecretvalue | kubeseal \
|
||||
--controller-namespace=kube-system \
|
||||
--controller-name=sealed-secrets-controller \
|
||||
--format=yaml \
|
||||
--name=my-secret \
|
||||
--namespace=default \
|
||||
--scope=strict
|
||||
|
||||
# If this works, plugin has a bug
|
||||
# If this fails too, controller/cluster issue
|
||||
```
|
||||
|
||||
### Compare Encryption
|
||||
|
||||
```bash
|
||||
# Encrypt same value with both methods
|
||||
# Plugin output:
|
||||
kubectl get sealedsecret my-secret-plugin -n default -o jsonpath='{.spec.encryptedData.password}'
|
||||
|
||||
# kubeseal output:
|
||||
echo -n mysecretvalue | kubeseal --raw --cert cert.pem --scope strict --name my-secret --namespace default
|
||||
|
||||
# Values should be different (encryption is random)
|
||||
# But both should unseal to same plaintext
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If encryption still fails:
|
||||
|
||||
1. **Gather diagnostics**:
|
||||
```bash
|
||||
# Controller version
|
||||
kubectl get deployment -n kube-system sealed-secrets-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
|
||||
|
||||
# Certificate validity
|
||||
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text
|
||||
|
||||
# Plugin version (in Headlamp UI)
|
||||
Settings → Sealed Secrets → About
|
||||
```
|
||||
|
||||
2. **Enable debug logging** (see above)
|
||||
|
||||
3. **Try kubeseal CLI** to isolate issue
|
||||
|
||||
4. **Report issue**:
|
||||
- [Plugin Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)
|
||||
- [Controller Issues](https://github.com/bitnami-labs/sealed-secrets/issues)
|
||||
|
||||
Include:
|
||||
- Browser and version
|
||||
- Plugin version
|
||||
- Controller version
|
||||
- Full error message
|
||||
- Browser console logs
|
||||
- Steps to reproduce
|
||||
|
||||
## See Also
|
||||
|
||||
- [Common Errors](common-errors.md) - General troubleshooting
|
||||
- [Controller Issues](controller-issues.md) - Controller-specific problems
|
||||
- [Encryption Flow](../architecture/encryption-flow.md) - How encryption works
|
||||
- [Creating Secrets](../user-guide/creating-secrets.md) - Usage guide
|
||||
@@ -0,0 +1,794 @@
|
||||
# Permission Errors
|
||||
|
||||
RBAC troubleshooting for Sealed Secrets operations.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Understanding RBAC](#understanding-rbac)
|
||||
- [Common Permission Errors](#common-permission-errors)
|
||||
- [Diagnosing Permission Issues](#diagnosing-permission-issues)
|
||||
- [Fixing Permissions](#fixing-permissions)
|
||||
- [Service Accounts](#service-accounts)
|
||||
- [Namespace-Scoped vs Cluster-Wide](#namespace-scoped-vs-cluster-wide)
|
||||
|
||||
---
|
||||
|
||||
## Understanding RBAC
|
||||
|
||||
The plugin requires different permissions for different operations:
|
||||
|
||||
| Operation | Required Permissions |
|
||||
|-----------|---------------------|
|
||||
| **View list** | `list` sealedsecrets.bitnami.com |
|
||||
| **View details** | `get` sealedsecrets.bitnami.com |
|
||||
| **Create** | `create` sealedsecrets.bitnami.com |
|
||||
| **Delete** | `delete` sealedsecrets.bitnami.com |
|
||||
| **Download cert** | `get` services or services/proxy |
|
||||
| **Decrypt** | `get` secrets |
|
||||
| **List namespaces** | `list` namespaces |
|
||||
|
||||
### How Plugin Checks Permissions
|
||||
|
||||
The plugin uses Kubernetes `SelfSubjectAccessReview` API to check permissions in real-time:
|
||||
|
||||
```bash
|
||||
# Example: Check if you can create SealedSecrets
|
||||
kubectl create -f - <<EOF
|
||||
apiVersion: authorization.k8s.io/v1
|
||||
kind: SelfSubjectAccessReview
|
||||
spec:
|
||||
resourceAttributes:
|
||||
group: bitnami.com
|
||||
resource: sealedsecrets
|
||||
verb: create
|
||||
namespace: default
|
||||
EOF
|
||||
```
|
||||
|
||||
**Plugin behavior**:
|
||||
- ✅ Permission granted → Show UI element
|
||||
- ❌ Permission denied → Hide/disable UI element
|
||||
- ⚠️ Check fails → Assume no permission (fail-safe)
|
||||
|
||||
---
|
||||
|
||||
## Common Permission Errors
|
||||
|
||||
### "Cannot list SealedSecrets"
|
||||
|
||||
#### Symptom
|
||||
|
||||
Empty list or error:
|
||||
```
|
||||
Failed to load sealed secrets: Forbidden (403)
|
||||
User 'alice' cannot list resource 'sealedsecrets' in API group 'bitnami.com'
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Missing `list` permission for sealedsecrets.bitnami.com.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check permission
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com --all-namespaces
|
||||
|
||||
# Check as specific user (requires admin)
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com --as alice
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
Apply viewer role (requires cluster-admin):
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-viewer
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-sealed-secrets-viewer
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-viewer
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Apply:
|
||||
```bash
|
||||
kubectl apply -f sealed-secrets-viewer.yaml
|
||||
|
||||
# Verify
|
||||
kubectl auth can-i list sealedsecrets.bitnami.com --as alice
|
||||
# Output: yes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Create button not showing"
|
||||
|
||||
#### Symptom
|
||||
|
||||
"Create Sealed Secret" button missing from UI.
|
||||
|
||||
#### Cause
|
||||
|
||||
Missing `create` permission for sealedsecrets.bitnami.com.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com -n production
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
Apply creator role:
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-creator
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list", "create"]
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-sealed-secrets-creator
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-creator
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Apply:
|
||||
```bash
|
||||
kubectl apply -f sealed-secrets-creator.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Cannot download certificate"
|
||||
|
||||
#### Symptom
|
||||
|
||||
Download button hidden or error:
|
||||
```
|
||||
Failed to fetch certificate: Forbidden (403)
|
||||
User cannot access service 'sealed-secrets-controller'
|
||||
```
|
||||
|
||||
#### Cause
|
||||
|
||||
Missing service access permission.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
# Check service access
|
||||
kubectl auth can-i get services -n kube-system
|
||||
kubectl auth can-i get services/sealed-secrets-controller -n kube-system
|
||||
|
||||
# Check proxy access
|
||||
kubectl auth can-i get services/proxy -n kube-system
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
**Option 1: Direct service access** (simpler):
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-cert-downloader
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-cert-downloader
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-cert-downloader
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
**Option 2: Proxy access** (if direct access doesn't work):
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-proxy-access
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["services/proxy"]
|
||||
verbs: ["get", "create"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
```
|
||||
|
||||
Apply:
|
||||
```bash
|
||||
kubectl apply -f cert-downloader.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Decrypt button hidden"
|
||||
|
||||
#### Symptom
|
||||
|
||||
"Decrypt" button not visible on SealedSecret detail page.
|
||||
|
||||
#### Cause
|
||||
|
||||
Missing `get` permission for plain Secrets.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
kubectl auth can-i get secrets
|
||||
kubectl auth can-i get secrets -n production
|
||||
```
|
||||
|
||||
#### Security Warning
|
||||
|
||||
⚠️ **Granting Secret access allows viewing all unsealed secrets!**
|
||||
|
||||
Only grant to authorized users (security team, ops).
|
||||
|
||||
#### Solution
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-decryptor
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get", "list"] # ⚠️ Sensitive permission
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: security-team-decryptor
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-decryptor
|
||||
subjects:
|
||||
- kind: Group
|
||||
name: security-team # Use groups, not individual users
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
**Namespace-scoped alternative** (more secure):
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: sealed-secrets-decryptor
|
||||
namespace: production # Only in production namespace
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get", "list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: alice-decryptor
|
||||
namespace: production
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: sealed-secrets-decryptor
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### "Delete button disabled"
|
||||
|
||||
#### Symptom
|
||||
|
||||
"Delete" button is grayed out or hidden.
|
||||
|
||||
#### Cause
|
||||
|
||||
Missing `delete` permission for sealedsecrets.bitnami.com.
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
```bash
|
||||
kubectl auth can-i delete sealedsecrets.bitnami.com
|
||||
kubectl auth can-i delete sealedsecrets.bitnami.com -n production
|
||||
```
|
||||
|
||||
#### Solution
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-admin
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list", "create", "delete"]
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get"]
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces"]
|
||||
verbs: ["list"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-sealed-secrets-admin
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-admin
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Diagnosing Permission Issues
|
||||
|
||||
### Check Your Current User
|
||||
|
||||
```bash
|
||||
# Who am I?
|
||||
kubectl auth whoami
|
||||
|
||||
# Or in older versions:
|
||||
kubectl config view --minify -o jsonpath='{.contexts[0].context.user}'
|
||||
```
|
||||
|
||||
### List All Your Permissions
|
||||
|
||||
```bash
|
||||
# List all permissions
|
||||
kubectl auth can-i --list
|
||||
|
||||
# List permissions in specific namespace
|
||||
kubectl auth can-i --list -n production
|
||||
```
|
||||
|
||||
### Check Specific Permission
|
||||
|
||||
```bash
|
||||
# Template:
|
||||
kubectl auth can-i <verb> <resource>.<apiGroup> [-n <namespace>]
|
||||
|
||||
# Examples:
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com
|
||||
kubectl auth can-i get secrets -n production
|
||||
kubectl auth can-i list namespaces
|
||||
```
|
||||
|
||||
### View Your Roles
|
||||
|
||||
```bash
|
||||
# View RoleBindings (namespace-scoped)
|
||||
kubectl get rolebindings -A -o json | \
|
||||
jq -r '.items[] | select(.subjects[]?.name == "'$(kubectl auth whoami)'")'
|
||||
|
||||
# View ClusterRoleBindings (cluster-wide)
|
||||
kubectl get clusterrolebindings -o json | \
|
||||
jq -r '.items[] | select(.subjects[]?.name == "'$(kubectl auth whoami)'")'
|
||||
```
|
||||
|
||||
### Check As Another User (Admin Only)
|
||||
|
||||
```bash
|
||||
# Check as specific user
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com --as alice
|
||||
|
||||
# Check as user in group
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com --as alice --as-group developers
|
||||
|
||||
# Check as service account
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com --as system:serviceaccount:ci-cd:github-actions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixing Permissions
|
||||
|
||||
### General Troubleshooting Steps
|
||||
|
||||
1. **Identify missing permission**:
|
||||
```bash
|
||||
kubectl auth can-i <verb> <resource>
|
||||
```
|
||||
|
||||
2. **Check existing roles**:
|
||||
```bash
|
||||
kubectl get clusterroles | grep sealed-secrets
|
||||
kubectl describe clusterrole <role-name>
|
||||
```
|
||||
|
||||
3. **Apply appropriate role** (see examples above)
|
||||
|
||||
4. **Verify permission granted**:
|
||||
```bash
|
||||
kubectl auth can-i <verb> <resource>
|
||||
# Should output: yes
|
||||
```
|
||||
|
||||
5. **Refresh Headlamp UI** (plugin re-checks permissions)
|
||||
|
||||
### Using Pre-Built Roles
|
||||
|
||||
See [RBAC Permissions Guide](../user-guide/rbac-permissions.md) for pre-built role examples:
|
||||
|
||||
- **Viewer** - Read-only access
|
||||
- **Creator** - Create and view
|
||||
- **Admin** - Full access (including delete)
|
||||
- **Auditor** - View sealed and unsealed secrets
|
||||
|
||||
---
|
||||
|
||||
## Service Accounts
|
||||
|
||||
### CI/CD Service Account
|
||||
|
||||
For automated systems (GitHub Actions, GitLab CI, etc.):
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: sealed-secrets-ci
|
||||
namespace: ci-cd
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: sealed-secrets-ci-creator
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["create", "get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get"]
|
||||
resourceNames: ["sealed-secrets-controller"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: sealed-secrets-ci
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-ci-creator
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: sealed-secrets-ci
|
||||
namespace: ci-cd
|
||||
```
|
||||
|
||||
Apply:
|
||||
```bash
|
||||
kubectl apply -f ci-service-account.yaml
|
||||
|
||||
# Get token (Kubernetes 1.24+)
|
||||
kubectl create token sealed-secrets-ci -n ci-cd --duration=8760h
|
||||
```
|
||||
|
||||
### Using Service Account in kubeseal
|
||||
|
||||
```bash
|
||||
# Get token
|
||||
TOKEN=$(kubectl create token sealed-secrets-ci -n ci-cd)
|
||||
|
||||
# Use with kubeseal
|
||||
echo -n mysecret | kubeseal \
|
||||
--controller-namespace=kube-system \
|
||||
--token="$TOKEN" \
|
||||
--format=yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Namespace-Scoped vs Cluster-Wide
|
||||
|
||||
### When to Use Role (Namespace-Scoped)
|
||||
|
||||
Use `Role` + `RoleBinding` when:
|
||||
- Users should only access secrets in specific namespaces
|
||||
- Following principle of least privilege
|
||||
- Team-based access (dev team → dev namespace)
|
||||
|
||||
Example:
|
||||
```yaml
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: sealed-secrets-creator
|
||||
namespace: development # Only in development namespace
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["create", "get", "list", "delete"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: dev-team-sealed-secrets
|
||||
namespace: development
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: sealed-secrets-creator
|
||||
subjects:
|
||||
- kind: Group
|
||||
name: developers
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
### When to Use ClusterRole
|
||||
|
||||
Use `ClusterRole` + `ClusterRoleBinding` when:
|
||||
- Users need access across all namespaces
|
||||
- Platform/SRE team
|
||||
- Shared services
|
||||
|
||||
Example: See [RBAC Permissions Guide](../user-guide/rbac-permissions.md)
|
||||
|
||||
### Combining Both
|
||||
|
||||
User can have:
|
||||
- ClusterRole for viewing (read all namespaces)
|
||||
- Role for creating (write only to specific namespace)
|
||||
|
||||
```yaml
|
||||
# ClusterRole: View all SealedSecrets
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: alice-viewer
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: sealed-secrets-viewer
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
|
||||
# Role: Create only in development
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: alice-creator
|
||||
namespace: development
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: sealed-secrets-creator
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Principle of Least Privilege
|
||||
|
||||
Grant minimum permissions needed:
|
||||
|
||||
```yaml
|
||||
# ✅ Good: Specific permissions
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list"] # Only read
|
||||
|
||||
# ❌ Bad: Wildcard permissions
|
||||
rules:
|
||||
- apiGroups: ["*"]
|
||||
resources: ["*"]
|
||||
verbs: ["*"] # Too permissive!
|
||||
```
|
||||
|
||||
### 2. Use Groups, Not Individual Users
|
||||
|
||||
```yaml
|
||||
# ✅ Good: Use groups
|
||||
subjects:
|
||||
- kind: Group
|
||||
name: developers
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
|
||||
# ❌ Bad: Individual users
|
||||
subjects:
|
||||
- kind: User
|
||||
name: alice
|
||||
- kind: User
|
||||
name: bob
|
||||
# Hard to maintain as team grows
|
||||
```
|
||||
|
||||
### 3. Separate Read and Write
|
||||
|
||||
```yaml
|
||||
# Viewer role: Read-only
|
||||
- verbs: ["get", "list"]
|
||||
|
||||
# Creator role: Create new
|
||||
- verbs: ["create"]
|
||||
|
||||
# Admin role: Full access
|
||||
- verbs: ["get", "list", "create", "update", "patch", "delete"]
|
||||
```
|
||||
|
||||
### 4. Limit Secret Access
|
||||
|
||||
```yaml
|
||||
# ✅ Good: Only SealedSecrets
|
||||
rules:
|
||||
- apiGroups: ["bitnami.com"]
|
||||
resources: ["sealedsecrets"]
|
||||
verbs: ["get", "list", "create"]
|
||||
# Cannot view unsealed Secrets
|
||||
|
||||
# ❌ Risky: Include Secret access
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get", "list"]
|
||||
# Can view all unsealed secrets!
|
||||
```
|
||||
|
||||
### 5. Audit RBAC Changes
|
||||
|
||||
```bash
|
||||
# List all ClusterRoleBindings for SealedSecrets
|
||||
kubectl get clusterrolebindings -o json | \
|
||||
jq '.items[] | select(.roleRef.name | contains("sealed-secrets"))'
|
||||
|
||||
# Audit who has Secret access
|
||||
kubectl get clusterrolebindings -o json | \
|
||||
jq '.items[] | select(.roleRef.name | contains("admin") or contains("edit"))'
|
||||
```
|
||||
|
||||
### 6. Regular Access Reviews
|
||||
|
||||
```bash
|
||||
# Who can create SealedSecrets?
|
||||
kubectl get rolebindings,clusterrolebindings -A -o json | \
|
||||
jq -r '.items[] | select(.roleRef.name | contains("sealed-secrets")) |
|
||||
"\(.metadata.name): \(.subjects[].name)"'
|
||||
|
||||
# Review quarterly and remove unnecessary permissions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Commands
|
||||
|
||||
```bash
|
||||
# Quick permission check
|
||||
kubectl auth can-i create sealedsecrets.bitnami.com && echo "✅ Can create" || echo "❌ Cannot create"
|
||||
|
||||
# List all sealed-secrets-related roles
|
||||
kubectl get clusterroles,roles -A | grep sealed-secrets
|
||||
|
||||
# Describe specific role
|
||||
kubectl describe clusterrole sealed-secrets-viewer
|
||||
|
||||
# View who has a role
|
||||
kubectl get clusterrolebindings sealed-secrets-viewer -o yaml
|
||||
|
||||
# Check if role exists
|
||||
kubectl get clusterrole sealed-secrets-creator &>/dev/null && echo "✅ Exists" || echo "❌ Not found"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If permission issues persist:
|
||||
|
||||
1. **Verify cluster-admin access**: Some fixes require cluster-admin
|
||||
```bash
|
||||
kubectl auth can-i '*' '*' --all-namespaces
|
||||
```
|
||||
|
||||
2. **Check with cluster administrator**: They may need to apply RBAC
|
||||
|
||||
3. **Review Kubernetes RBAC docs**:
|
||||
- [RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
|
||||
- [RBAC Good Practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/)
|
||||
|
||||
4. **Report plugin issues**:
|
||||
- [GitHub Issues](https://github.com/cpfarhood/headlamp-sealed-secrets-plugin/issues)
|
||||
|
||||
## See Also
|
||||
|
||||
- [RBAC Permissions Guide](../user-guide/rbac-permissions.md) - Complete RBAC documentation
|
||||
- [Common Errors](common-errors.md) - General troubleshooting
|
||||
- [Security Hardening](../deployment/security-hardening.md) - Production security
|
||||
Reference in New Issue
Block a user