Kubernetes GitOps Encryption

Overview

Encrypted GitOps refers to the practice of managing infrastructure and application deployments using GitOps principles, while ensuring that sensitive data (e.g., secrets, keys, credentials, or sensitive configuration) is securely encrypted. GitOps is a workflow that uses Git as the single source of truth for declarative infrastructure and application definitions. In encrypted GitOps, the sensitive information is encrypted to ensure security when storing and using it as part of the GitOps pipeline.

This guide provides a step-by-step walkthrough to automate the setup of a secrets configuration for Flux in a Kubernetes cluster. The goal is to securely manage Docker Hub pull secrets using SOPS encryption and Flux GitOps.

ATTENTION! You should always check any committed file which might contain secrets, even if you think you’ve already encrypted it. This is one of the risks with using in-repository secret-keeping.

Key Steps

Prerequisites: Install SOPS, generate and export/import keys, define encryption rules
Create a Kubernetes Secret: Configure Docker Hub pull secrets using Kubernetes
Encrypt with SOPS: Securely encrypt the secret for safe storage in Git
Generate Kustomization Files: Enable Flux to manage the pull secret
Commit Changes to Git: Save your changes to the Git repository Flux uses
Trigger Flux Reconciliation (Optional): Force Flux to sync the changes immediately

Prerequisites

Install SOPS

Install SOPS, a tool used for encrypting secrets:

SOPS_LATEST_VERSION=$(curl -s "https://api.github.com/repos/getsops/sops/releases/latest" | grep -Po '"tag_name": "v\K[0-9.]+')
curl -Lo sops.deb "https://github.com/getsops/sops/releases/download/v${SOPS_LATEST_VERSION}/sops_${SOPS_LATEST_VERSION}_amd64.deb"
sudo apt --fix-broken install ./sops.deb

Generate GPG Key

Define the Flux and your Personal key names and emails. Some files you want to be decrypted by Flux and you, some – only by you:

export CLUSTER_KEY_NAME="flux-secrets.cluster.local"
export CLUSTER_KEY_EMAIL="[email protected]"
export KEY_NAME="User Name"
export KEY_EMAIL="[email protected]"

Generate the GPG keys:

gpgconf --kill gpg-agent
gpgconf --launch gpg-agent
# Generate the Flux key
gpg --batch --expert --pinentry-mode loopback --gen-key <<EOF
%echo Generating an ECC OpenPGP key
%no-protection
Key-Type: eddsa
Key-Curve: ed25519
Key-Usage: sign,cert
Subkey-Type: ecdh
Subkey-Curve: cv25519
Expire-Date: 0
Name-Real: ${CLUSTER_KEY_NAME}
Name-Email: ${CLUSTER_KEY_EMAIL}
Name-Comment: flux-secrets key
%commit
%echo done
EOF

# Generate the personal key 
# If you already have one, you may skip this step
gpgconf --kill gpg-agent
gpgconf --launch gpg-agent
gpg --batch --expert --pinentry-mode loopback --gen-key <<EOF
%echo Generating an ECC OpenPGP key
%no-protection
Key-Type: eddsa
Key-Curve: ed25519
Subkey-Type: ecdh
Subkey-Curve: cv25519
Expire-Date: 0
Name-Real: ${KEY_NAME}
Name-Email: ${KEY_EMAIL}
Name-Comment: personal key
%commit
%echo done
EOF

List generated keys:

# Sample output:
#
# /home/<...>/.gnupg/pubring.kbx
# ------------------------------------
# pub   ed25519 2025-01-13 [SC]
#       8FEAD29979491490886352C63A7017716A9F6CA4
# uid           [ultimate] flux-secrets.cluster.local (flux secrets) <flux-secrets.cluster.local>
# sub   cv25519 2025-01-13 [E]
# pub   ed25519 2025-01-13 [SCA]
#       7FFF3DAD1485DA1E62FA71BCF51F012315119CC1
# uid           [ultimate] User Name(personal key) <[email protected]>
# sub   cv25519 2025-01-13 [E]

gpg --list-keys

Create the sops-gpg secret in the flux-system namespace by exporting/importing the Flux key:

gpg --export-secret-keys --armor \
"flux-secrets.cluster.local" | \
kubectl -n flux-system create secret \
generic sops-gpg --from-file=sops.asc=/dev/stdin

Export the public key for encryption, so anyone can add encrypted secrets to your repo:

gpg --export --armor "flux-secrets.cluster.local" > .flux.pub.asc

Define Encryption Rules

Create file with encryption rules in the root of your repository:

# .sops.yaml
---
# here 8FEA... (aka "flux-secrets.cluster.local") is Flux's public key 
# and 7FFF... (aka "User Name") is your public key.
# So any yaml files created with sops in git repository
# will be encrypted in a way that both Flux and you can decrypt them.
creation_rules:
  #
  # Ensure that talosconfig, kubeconfig and secrets.yaml all get encrypted 
  # using only your own public key, since we don't need nor want Flux
  # to be able to decrypt them
  #
  # Ensure files in the ./machineconfigs directory are encrypted specifically
  - path_regex: machineconfigs/.*\.yaml
    encrypted_regex: ^(secret|bootstraptoken|secretboxEncryptionSecret|token|key)$
    pgp: 7FFF3DAD1485DA1E62FA71BCF51F012315119CC1
  # Encrypt `talosconfig` files using only your public key
  - path_regex: talosconfig
    encrypted_regex: ^key$
    pgp: 7FFF3DAD1485DA1E62FA71BCF51F012315119CC1
  # Encrypt `kubeconfig` files using only your public key
  - path_regex: kubeconfig
    encrypted_regex: ^client-key-data$
    pgp: 7FFF3DAD1485DA1E62FA71BCF51F012315119CC1
  # Encrypt `secrets.yaml` using only your public key
  - path_regex: secrets.yaml
    encrypted_regex: ^(secret|bootstraptoken|secretboxencryptionsecret|token|key)$
    pgp: 7FFF3DAD1485DA1E62FA71BCF51F012315119CC1
  # Default rule for all other `.yaml` files in the repository
  - path_regex: ./.*\.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: >-
      8FEAD29979491490886352C63A7017716A9F6CA4,
      7FFF3DAD1485DA1E62FA71BCF51F012315119CC1

Optional: Encrypt and Add to Repo Sensitive Configuration Files

In the root of the repo folder:

# kubeconfig
talosctl -n $CONTROL_PLANE_IP kubeconfig kubeconfig -f
sops -e -i --input-type yaml --output-type yaml kubeconfig

# copy secrets.yaml created during cluster initialisation here
cp ~/talos-1/secrets.yaml .
sops -e -i secrets.yaml

# talosconfig
cp ~/.talos/config talosconfig
sops -e -i --input-type yaml --output-type yaml talosconfig

STOP! Make sure sensitive parts of the files are encrypted and only after that add them to the repo:

git add -A
git commit -m "Encrypted configuration files"
git push

Step 1: Create a Kubernetes Secret

Navigate to the root of your repository in the file system.

Define environment variables:

export SECRET_NAME=docker-hub-pull-secret
export SECRET_NAMESPACE=default
export DOCKER_REGISTRY_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<your username>  # Replace with your Docker Hub username.
export DOCKER_PASSWORD=<your password>  # Replace with your Docker Hub password.

Generate the Kubernetes secret:

kubectl create secret docker-registry \
  --dry-run=client \
  --namespace=$SECRET_NAMESPACE \
  --docker-server=$DOCKER_REGISTRY_SERVER \
  --docker-username=$DOCKER_USERNAME \
  --docker-password=$DOCKER_PASSWORD \
  $SECRET_NAME \
  -o yaml > infrastructure/security/pull-secrets/docker-hub.yaml

Encrypt the secret:

sops -e -i infrastructure/security/pull-secrets/docker-hub.yaml

Verify the encryption:

cat infrastructure/security/pull-secrets/docker-hub.yaml

Step 2: Create Kustomization Files

Add a kustomization.yaml file:

cat <<EOL > infrastructure/security/pull-secrets/kustomization.yaml
# File: infrastructure/security/pull-secrets/kustomization.yaml
# This file defines the pull-secrets Kustomization used to manage secrets in the Flux system
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: pull-secrets
  namespace: flux-system
resources:
- docker-hub.yaml
EOL

Add a new Kustomization for Flux:

cat <<EOL > clusters/production/flux-system/pull-secrets.yaml
# File: clusters/production/flux-system/pull-secrets.yaml
# This Kustomization defines how Flux manages the pull-secrets in the flux-system namespace
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: pull-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure/security/pull-secrets
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
EOL

Step 3: Commit and Push Changes

Add the files to Git:

git add -A

Commit and push:

git commit -m "Add pull-secrets Kustomization for Flux"
git push origin main

Step 4: Reconcile Flux

Flux reconciles changes automatically at set intervals (e.g., every 10 minutes). To trigger reconciliation immediately:

Reconcile the flux-system Kustomization:

flux reconcile kustomization flux-system --namespace flux-system

Reconcile pull-secrets:

flux reconcile kustomization pull-secrets --namespace flux-system

Step 5: Verify Deployment

Verify the secret is deployed:

kubectl -n default get secrets docker-hub-pull-secret

Check Flux logs if issues arise:

kubectl -n flux-system logs deploy/kustomize-controller

Scripted Steps 1-5

Find below the script with the key steps 1-5 above and some health checks for debugging purposes.

Update it with your Docker.com username and password or export them before running the script.

#!/usr/bin/env bash
# This script automates the setup of a pull-secrets 
# configuration for Flux in a Kubernetes cluster.
# It creates a Docker Hub pull secret, encrypts it with SOPS, 
# adds it to Git repo
# and configures Flux to manage it.
# Key steps:
# 1. Create a Kubernetes secret for Docker Hub credentials.
# 2. Encrypt the secret with SOPS for secure storage.
# 3. Generate Kustomization files to enable Flux to manage the secret.
# 4. Commit and push changes to the Git repository.
# 5. Trigger Flux to reconcile and deploy the pull secret. This step is optional
# as Flux does reconcile every N minutes configured in pull-secrets.yaml

export SECRET_NAME=docker-hub-pull-secret
export SECRET_NAMESPACE=default
export DOCKER_REGISTRY_SERVER=https://index.docker.io/v1/
export DOCKER_USERNAME=<your username> # Replace with your Docker Hub username.
export DOCKER_PASSWORD=<your password> # Replace with your Docker Hub password.

# Exit immediately on errors or unset variables, and pipe failures
set -euo pipefail

##################################################
# 0. Optional: Sanity checks before we begin
##################################################

# Verify we are in a Git repo
if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  echo "[ERROR] Not inside a valid Git repository. Please run in your repo root."
  exit 1
fi

# Function to check command execution
check_command() {
  if [ $? -ne 0 ]; then
    echo "Error: $1 failed. Exiting script."
    exit 1
  fi
}

# Function to test if SOPS encryption was successful
test_sops_encryption() {
  if grep -q "ENC\[" "$1"; then
    echo "Success: SOPS encrypted sensitive data in $1"
  else
    echo "Error: SOPS did not encrypt sensitive data in $1. Exiting script."
    exit 1
  fi
}

# Function to log Flux reconciliation output
log_flux_reconcile() {
  local output
  output=$(flux reconcile kustomization $1 --namespace flux-system 2>&1)
  echo "$output"
  if [[ "$output" == *"error"* ]]; then
    echo "Error during reconciliation: $output"
    exit 1
  fi
}

# Validate pull-secrets.yaml file
validate_pull_secrets_kustomization() {
  if [ ! -f "clusters/production/flux-system/pull-secrets.yaml" ]; then
    echo "Error: pull-secrets.yaml not found in clusters/production/flux-system. Exiting script."
    exit 1
  fi
}

# Step 1: Create pull-secrets directory and secret configuration
echo "Setting up pull-secrets configuration under infrastructure/security..."
mkdir -p infrastructure/security/pull-secrets
check_command "Creating directory infrastructure/security/pull-secrets"

kubectl create secret docker-registry \
  --dry-run=client \
  --namespace=$SECRET_NAMESPACE \
  --docker-server=$DOCKER_REGISTRY_SERVER \
  --docker-username=$DOCKER_USERNAME \
  --docker-password=$DOCKER_PASSWORD \
  $SECRET_NAME \
  -o yaml > infrastructure/security/pull-secrets/docker-hub.yaml
check_command "Generating docker-hub-pull-secret in dry-run mode"

# Encrypt the secret
sops -e -i infrastructure/security/pull-secrets/docker-hub.yaml
check_command "Encrypting docker-hub-pull-secret with SOPS"

# Test if SOPS encryption was successful
test_sops_encryption infrastructure/security/pull-secrets/docker-hub.yaml

echo "Encrypted secret created in infrastructure/security/pull-secrets/docker-hub.yaml"

# Step 2: Create kustomization.yaml for pull-secrets
# Add a comment to the generated kustomization.yaml for pull-secrets
cat <<EOL > infrastructure/security/pull-secrets/kustomization.yaml
# File: infrastructure/security/pull-secrets/kustomization.yaml
# This file defines the pull-secrets Kustomization used to manage secrets in the Flux system
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: pull-secrets
  namespace: flux-system
resources:
- docker-hub.yaml
EOL
check_command "Creating kustomization.yaml for pull-secrets"

echo "kustomization.yaml created in infrastructure/security/pull-secrets"

# Step 3: Create pull-secrets Kustomization in Flux
# Add a comment to the generated Kustomization for Flux pull-secrets
cat <<EOL > clusters/production/flux-system/pull-secrets.yaml
# File: clusters/production/flux-system/pull-secrets.yaml
# This Kustomization defines how Flux manages the pull-secrets in the flux-system namespace
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: pull-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./infrastructure/security/pull-secrets
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
EOL
check_command "Creating pull-secrets.yaml in clusters/production/flux-system"

echo "pull-secrets.yaml created in clusters/production/flux-system"

# Step 4: Commit and Push Changes
git add infrastructure/security/pull-secrets clusters/production/flux-system/pull-secrets.yaml
check_command "Staging files for git commit"
git commit -m "Add pull-secrets Kustomization for Flux"
check_command "Committing changes to git"
git push origin main
check_command "Pushing changes to remote repository"

echo "Changes committed and pushed to repository."
cd ..
# Step 5: Reconcile flux-system and pull-secrets Kustomizations
echo "Reconciling flux-system Kustomization..."
log_flux_reconcile flux-system

# Wait for pull-secrets Kustomization to register
echo "Waiting for pull-secrets Kustomization to be registered..."
for i in {1..20}; do
  if flux get kustomizations -n flux-system | grep -q pull-secrets; then
    echo "pull-secrets Kustomization registered."
    break
  fi
  echo "Waiting for pull-secrets Kustomization to appear... ($i/20)"
  sleep 5
done

if ! flux get kustomizations -n flux-system | grep -q pull-secrets; then
  echo "Error: pull-secrets Kustomization not found after waiting. Exiting script."
  kubectl -n flux-system describe kustomization flux-system
  exit 1
fi

echo "Reconciling pull-secrets Kustomization..."
log_flux_reconcile pull-secrets

# Step 6: Verify deployment
echo "Verifying deployment of docker-hub-pull-secret..."
kubectl -n $SECRET_NAMESPACE get secrets docker-hub-pull-secret
check_command "Verifying deployed secret in namespace $SECRET_NAMESPACE"

Overview#

Key Steps#

Prerequisites#

Install SOPS#

Generate GPG Key#

Define Encryption Rules#

Optional: Encrypt and Add to Repo Sensitive Configuration Files#

Step 1: Create a Kubernetes Secret#

Step 2: Create Kustomization Files#

Step 3: Commit and Push Changes#

Step 4: Reconcile Flux#

Step 5: Verify Deployment#

Scripted Steps 1-5#