headlamp-sealed-secrets-plugin/docs/tutorials/ci-cd-integration.md

CI/CD Integration Tutorial

Learn how to automate sealed secret creation in your CI/CD pipelines.

Overview

This tutorial shows you how to:

• Create sealed secrets in CI/CD pipelines
• Download sealing certificates for offline encryption
• Use kubeseal CLI with plugin-exported certificates
• Integrate with GitHub Actions, GitLab CI, and Jenkins

Prerequisites

• Headlamp Sealed Secrets plugin installed
• Sealed Secrets controller running in your cluster
• Access to download sealing certificates
• CI/CD system (GitHub Actions, GitLab CI, or Jenkins)

Step 1: Download the Sealing Certificate

The sealing certificate is the public key used to encrypt secrets. You can download it from Headlamp:

Using Headlamp UI

1. Navigate to Sealed Secrets → Sealing Keys
2. Find the active certificate (no expiry warning)
3. Click Download
4. Save as sealed-secrets-cert.pem

Using kubectl

Alternatively, fetch it directly from the controller:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d > sealed-secrets-cert.pem

Or use the controller's certificate endpoint:

curl http://sealed-secrets-controller.kube-system:8080/v1/cert.pem > sealed-secrets-cert.pem

Step 2: Install kubeseal CLI

Install the kubeseal command-line tool:

macOS (Homebrew):

brew install kubeseal

Linux:

KUBESEAL_VERSION='0.24.0'
wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Windows (Chocolatey):

choco install kubeseal

Verify installation:

kubeseal --version
# Output: kubeseal version: v0.24.0

Step 3: Create Sealed Secrets in CI/CD

GitHub Actions Example

Create .github/workflows/sealed-secrets.yml:

name: Create Sealed Secrets

on:
  push:
    paths:
      - 'secrets/**'
  workflow_dispatch:

jobs:
  seal-secrets:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install kubeseal
        run: |
          KUBESEAL_VERSION='0.24.0'
          wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
          tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
          sudo install -m 755 kubeseal /usr/local/bin/kubeseal

      - name: Download sealing certificate
        run: |
          # Option 1: From repository secret
          echo "${{ secrets.SEALED_SECRETS_CERT }}" > sealed-secrets-cert.pem

          # Option 2: From cluster (requires kubectl access)
          # kubectl get secret -n kube-system \
          #   -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
          #   -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d > sealed-secrets-cert.pem

      - name: Create sealed secret
        run: |
          # Create a plain Kubernetes secret
          kubectl create secret generic my-app-secret \
            --from-literal=database-password=${{ secrets.DB_PASSWORD }} \
            --from-literal=api-key=${{ secrets.API_KEY }} \
            --dry-run=client \
            -o yaml > secret.yaml

          # Seal the secret
          kubeseal --cert sealed-secrets-cert.pem \
            --format=yaml < secret.yaml > sealed-secret.yaml

          # Commit and push (optional)
          git add sealed-secret.yaml
          git commit -m "chore: update sealed secret"
          git push

      - name: Apply to cluster
        run: |
          kubectl apply -f sealed-secret.yaml

Store the certificate as a GitHub Secret:

1. Go to repository Settings → Secrets and variables → Actions
2. Click "New repository secret"
3. Name: SEALED_SECRETS_CERT
4. Value: Paste contents of sealed-secrets-cert.pem

GitLab CI Example

Create .gitlab-ci.yml:

stages:
  - seal
  - deploy

variables:
  KUBESEAL_VERSION: "0.24.0"

seal-secrets:
  stage: seal
  image: alpine:latest
  before_script:
    - apk add --no-cache curl tar
    - curl -LO "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
    - tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
    - mv kubeseal /usr/local/bin/
    - chmod +x /usr/local/bin/kubeseal
  script:
    # Get certificate from GitLab CI variable
    - echo "$SEALED_SECRETS_CERT" > sealed-secrets-cert.pem

    # Create and seal secret
    - |
      cat <<EOF > secret.yaml
      apiVersion: v1
      kind: Secret
      metadata:
        name: my-app-secret
        namespace: production
      stringData:
        database-password: "${DB_PASSWORD}"
        api-key: "${API_KEY}"
      EOF

    - kubeseal --cert sealed-secrets-cert.pem --format=yaml < secret.yaml > sealed-secret.yaml
  artifacts:
    paths:
      - sealed-secret.yaml
  only:
    - main

deploy-sealed-secret:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl apply -f sealed-secret.yaml
  dependencies:
    - seal-secrets
  only:
    - main

Set GitLab CI Variables:

1. Go to Settings → CI/CD → Variables
2. Add SEALED_SECRETS_CERT (type: File)
3. Add DB_PASSWORD and API_KEY (type: Masked)

Jenkins Pipeline Example

Create Jenkinsfile:

pipeline {
    agent any

    environment {
        KUBESEAL_VERSION = '0.24.0'
        NAMESPACE = 'production'
    }

    stages {
        stage('Install kubeseal') {
            steps {
                sh '''
                    if ! command -v kubeseal &> /dev/null; then
                        wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
                        tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
                        sudo install -m 755 kubeseal /usr/local/bin/kubeseal
                    fi
                '''
            }
        }

        stage('Download Certificate') {
            steps {
                withCredentials([file(credentialsId: 'sealed-secrets-cert', variable: 'CERT_FILE')]) {
                    sh 'cp $CERT_FILE sealed-secrets-cert.pem'
                }
            }
        }

        stage('Create Sealed Secret') {
            steps {
                withCredentials([
                    string(credentialsId: 'db-password', variable: 'DB_PASSWORD'),
                    string(credentialsId: 'api-key', variable: 'API_KEY')
                ]) {
                    sh '''
                        # Create secret manifest
                        kubectl create secret generic my-app-secret \
                            --namespace=${NAMESPACE} \
                            --from-literal=database-password=${DB_PASSWORD} \
                            --from-literal=api-key=${API_KEY} \
                            --dry-run=client \
                            -o yaml > secret.yaml

                        # Seal it
                        kubeseal --cert sealed-secrets-cert.pem \
                            --format=yaml < secret.yaml > sealed-secret.yaml

                        # Show sealed secret (safe to log)
                        cat sealed-secret.yaml
                    '''
                }
            }
        }

        stage('Deploy') {
            steps {
                sh 'kubectl apply -f sealed-secret.yaml'
            }
        }
    }

    post {
        always {
            cleanWs()
        }
    }
}

Step 4: Verify Sealed Secret

After creating the sealed secret, verify it was created and unsealed:

# Check sealed secret exists
kubectl get sealedsecret my-app-secret -n production

# Check the unsealed secret was created
kubectl get secret my-app-secret -n production

# Verify secret contains correct keys
kubectl get secret my-app-secret -n production -o jsonpath='{.data}' | jq 'keys'
# Output: ["api-key", "database-password"]

Advanced Patterns

Pattern 1: Different Secrets per Environment

Create environment-specific sealed secrets:

# Development
kubectl create secret generic my-app-secret \
  --namespace=dev \
  --from-literal=api-url=https://dev-api.example.com \
  --dry-run=client -o yaml | \
kubeseal --cert sealed-secrets-cert.pem --format=yaml > dev-sealed-secret.yaml

# Production
kubectl create secret generic my-app-secret \
  --namespace=production \
  --from-literal=api-url=https://api.example.com \
  --dry-run=client -o yaml | \
kubeseal --cert sealed-secrets-cert.pem --format=yaml > prod-sealed-secret.yaml

Pattern 2: Bulk Secret Creation

Create multiple secrets from a directory:

#!/bin/bash
# seal-all-secrets.sh

CERT="sealed-secrets-cert.pem"
SECRETS_DIR="plain-secrets"
OUTPUT_DIR="sealed-secrets"

mkdir -p "$OUTPUT_DIR"

for secret_file in "$SECRETS_DIR"/*.yaml; do
    filename=$(basename "$secret_file")
    kubeseal --cert "$CERT" --format=yaml < "$secret_file" > "$OUTPUT_DIR/$filename"
    echo "Sealed: $filename"
done

Pattern 3: Update Existing Sealed Secret

To update a sealed secret, re-seal it completely:

# Get current secret
kubectl get sealedsecret my-app-secret -o yaml > current-sealed.yaml

# Extract metadata
# ... modify as needed ...

# Create new version
kubectl create secret generic my-app-secret \
  --from-literal=database-password=NEW_PASSWORD \
  --from-literal=api-key=NEW_KEY \
  --dry-run=client -o yaml | \
kubeseal --cert sealed-secrets-cert.pem --format=yaml > updated-sealed-secret.yaml

# Apply
kubectl apply -f updated-sealed-secret.yaml

Best Practices

1. Certificate Management

✅ DO:

• Store certificate in CI/CD secrets (encrypted at rest)
• Download fresh certificate periodically (before expiry)
• Use certificate from same cluster where secrets will be deployed

❌ DON'T:

• Commit certificate to Git (it's public, but still clutters repo)
• Use expired certificates
• Mix certificates from different clusters

2. Secret Rotation

# Check certificate expiry in Headlamp
# Sealed Secrets → Sealing Keys → Check "Valid Until"

# Download new certificate before expiry
# Re-seal all secrets with new certificate
# Deploy new sealed secrets

3. Scope Selection

• Use strict scope for production secrets:

  kubeseal --cert cert.pem --scope strict < secret.yaml
  

• Use namespace-wide for shared secrets:

  kubeseal --cert cert.pem --scope namespace-wide < secret.yaml
  

• Use cluster-wide only for truly global secrets:

  kubeseal --cert cert.pem --scope cluster-wide < secret.yaml
  

4. GitOps Integration

Store sealed secrets in Git alongside manifests:

my-app/
├── deployment.yaml
├── service.yaml
└── sealed-secret.yaml  # Safe to commit!

Apply with:

kubectl apply -f my-app/

Troubleshooting

"no key could decrypt secret"

Cause: Certificate mismatch or secret was sealed for different namespace/name.

Solution:

# Verify you're using the correct certificate
kubeseal --cert sealed-secrets-cert.pem --validate < sealed-secret.yaml

# Re-seal with correct scope and metadata
kubectl create secret generic EXACT_NAME \
  --namespace EXACT_NAMESPACE \
  --from-literal=key=value \
  --dry-run=client -o yaml | \
kubeseal --cert sealed-secrets-cert.pem --scope strict --format=yaml > sealed-secret.yaml

"certificate has expired"

Solution: Download fresh certificate from Headlamp:

# Check expiry
openssl x509 -in sealed-secrets-cert.pem -noout -enddate

# Download new one from Headlamp UI or kubectl

CI/CD pipeline fails to seal

Check:

1. kubeseal is installed: kubeseal --version
2. Certificate file exists and is valid
3. Input secret YAML is well-formed
4. Namespace exists in cluster

Next Steps

• Multi-Cluster Setup - Manage secrets across multiple clusters
• Secret Rotation - Rotate secrets and certificates
• RBAC Permissions - Configure access control

Resources

• kubeseal CLI Documentation
• GitHub Actions Secrets
• GitLab CI Variables
• Jenkins Credentials