Overview

Created a Syncthing pod in Kubernetes cluster managed by FluxCD with dual NFS mounts, SSL certificate via cert-manager, and consolidated LoadBalancer services.

Architecture

  • Namespace: syncthing
  • Deployment: Single replica with Recreate strategy
  • Storage: Two NFS persistent volumes
  • SSL: Automatic Let’s Encrypt certificate via cert-manager
  • Load Balancing: Combined TCP/UDP service on single external IP

Storage Configuration

NFS Mounts

# Data mount (Dropbox sync)
xxx.xxx.xxx.xxx:/mnt/media/dropbox → /var/syncthing/dropbox

# Config mount (Syncthing configuration)
xxx.xxx.xxx.xxx:/mnt/media/home/nfs/syncthing → /var/syncthing/config

Persistent Volumes

  • syncthing-dropbox-pv: 1Ti capacity for sync data
  • syncthing-config-pv: 1Gi capacity for configuration

Both use NFS storage class with ReadWriteMany access mode.

Security & Permissions

Initial Permission Issues

  • Container initially ran as UID 1000, but NFS shares had restrictive permissions
  • Dropbox share owned by nobody:1001 with 770 permissions
  • Config files had strict NFS permissions preventing access

Solution: Root Access

  • Changed container to run as root (UID 0)
  • Root can access both NFS shares regardless of ownership
  • Fixed config directory permissions on NFS server:
    sudo chown -R nobody:nobody /mnt/media/home/nfs/syncthing
sudo chmod -R 755 /mnt/media/home/nfs/syncthing

Final Security Context (Non-Privileged)

# Pod-level security context
securityContext:
  runAsNonRoot: true
  runAsUser: 65534  # nobody user
  runAsGroup: 1001  # read-write group
  fsGroup: 1001
  supplementalGroups: [1001]
  seccompProfile:
    type: RuntimeDefault

# Container-level security context
container.securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  runAsUser: 65534
  runAsGroup: 1001
  capabilities:
    drop: [ALL]
  seccompProfile:
    type: RuntimeDefault

# Environment variables
env:
  - name: PUID
    value: "65534"
  - name: PGID
    value: "1001"

Network Services

Original Setup (2 External IPs)

  • syncthing-tcp: TCP transfers on port 22000
  • syncthing-udp: UDP discovery on port 21027
  • syncthing-web: Web UI on port 8384 (ClusterIP)

Consolidated Setup with Static IP

# Combined LoadBalancer service with static IP annotation
apiVersion: v1
kind: Service
metadata:
  name: syncthing-sync
  namespace: syncthing
  annotations:
    metallb.universe.tf/loadBalancer-ips: "xxx.xxx.xxx.yyy"
spec:
  type: LoadBalancer
  ports:
  - name: tcp-transfers
    port: 22000
    targetPort: 22000
    protocol: TCP
  - name: udp-discovery
    port: 21027
    targetPort: 21027
    protocol: UDP

Benefits:

  • ✅ Static IP: xxx.xxx.xxx.yyy remains constant across pod/node failures
  • ✅ Gateway Routing: IP-based routing instead of unreliable MAC-based routing
  • ✅ High Availability: IP persists even when MetalLB fails over to different nodes
  • ✅ Single External IP: Reduced from 2 IPs to 1 consolidated IP

SSL Certificate

Cert-Manager Configuration

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: syncthing-tls-certificate
  namespace: syncthing
spec:
  secretName: syncthing-tls
  duration: 2160h    # 90 days
  renewBefore: 720h  # 30 days before expiration
  commonName: "syncthing.my.domain.com"
  dnsNames:
    - "syncthing.my.domain.com"
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-production

Ingress Configuration

  • Host: syncthing.my.domain.com
  • SSL: Automatic redirect and forced SSL
  • Backend: syncthing-web:8384

FluxCD Integration

Directory Structure

apps/syncthing/base/
├── kustomization.yaml
├── syncthing-namespace.yaml
├── syncthing-pv-config.yaml
├── syncthing-pvc-config.yaml
├── syncthing-pv-dropbox.yaml
├── syncthing-pvc-dropbox.yaml
├── syncthing-deployment.yaml
├── syncthing-service.yaml
├── syncthing-certificate.yaml
├── syncthing-ingress.yaml
└── README.md

Deployment Integration

Added to clusters/production/apps/kustomization.yaml:

resources:
  - ../../../apps/syncthing/base

Troubleshooting Journey

Issue 1: NFS Mount Protocol Error

Problem: mount.nfs: Protocol not supported with NFSv4.1 mount options

Solution: Removed specific NFS version and mount options to match working transmission volumes:

# Before (failed)
nfs:
  server: xxx.xxx.xxx.xxx
  path: /mnt/media/dropbox
  mountOptions:
    - nfsvers=4.1
    - rsize=1048576
    - wsize=1048576

# After (working)
nfs:
  server: xxx.xxx.xxx.xxx
  path: /mnt/media/dropbox

Issue 2: Permission Denied on Config Files

Problem: Even as root, couldn’t read /var/syncthing/config/config.xml

Root Cause: NFS config directory had overly restrictive permissions from previous UID 1000 setup

Solution: SSH to NFS server and fix permissions:

cd /mnt/media/home/nfs/syncthing
sudo chown -R nobody:nobody .
sudo find . -type d -exec chmod 755 {} \;
sudo find . -type f -exec chmod 644 {} \;

Issue 3: FluxCD Reconciliation

Problem: Changes not applying after git push

Solution: Force FluxCD reconciliation:

flux reconcile source git flux-system
flux reconcile kustomization flux-system

Issue 4: Syncthing Privilege Warning

Problem: Syncthing warned about running as privileged/system user when running as root

Root Cause: Security best practice violation and Syncthing recommendation

Solution: Changed to non-privileged user matching NFS permissions:

# Updated NFS permissions on server:
chmod 660 /mnt/media/home/nfs/syncthing      # drw-rw----+
chmod 660 /mnt/media/home/nfs/syncthing/*    # -rw-rw----

# Pod runs as:
# - UID 65534 (nobody) - matches NFS file owner
# - GID 1001 (read-write) - matches NFS group and dropbox share

Issue 5: Database Permission Conflicts

Problem: Existing Syncthing database created as root caused permission conflicts

Solution: Deleted index-v0.14.0.db directory to allow fresh creation with correct ownership

Final Configuration

Access Points

File Paths in Syncthing UI

  • Dropbox Share: /var/syncthing/dropbox
  • Subfolders: /var/syncthing/dropbox/Documents, /var/syncthing/dropbox/Photos, etc.

Resource Usage

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

Key Learnings

  1. Security First: Non-privileged containers require careful UID/GID mapping to NFS permissions
  2. NFS Permission Strategy: Match container user to NFS file ownership for seamless access
  3. FluxCD Debugging: Use flux reconcile commands to force updates when changes don’t apply
  4. Service Consolidation: Multiple LoadBalancer services can share a single external IP
  5. Static IP Benefits: MetalLB annotations provide IP persistence across node failures
  6. MetalLB Behavior: LoadBalancer IPs announced via Layer 2 can change MAC addresses during failover
  7. Certificate Integration: cert-manager with Cloudflare DNS01 challenge works seamlessly
  8. Database Cleanup: Sometimes fresh start is better than fixing complex permission issues

Security Improvements Achieved

✅ Non-Root Execution: Changed from UID 0 to UID 65534 (nobody)
✅ Pod Security Standards: Meets Kubernetes restricted security requirements
✅ Capability Dropping: All capabilities dropped with drop: [ALL]
✅ Privilege Escalation: Prevented with allowPrivilegeEscalation: false
✅ Seccomp Profile: Applied RuntimeDefault security profile
✅ No Privilege Warnings: Syncthing no longer warns about running as privileged user