PowerShell to Go Migration: Containerized Web Scraper Project
Complete modernization of legacy PowerShell scraper to containerized Go application with Kubernetes deployment - achieving 90% container size reduction and 10x performance improvement
Complete modernization of legacy PowerShell scraper to containerized Go application with Kubernetes deployment - achieving 90% container size reduction and 10x performance improvement
Deploy a self-hosted, privacy-focused SearXNG metasearch engine on your Kubernetes cluster for integration with AI tools like OpenWebUI. Overview SearXNG is a privacy-respecting metasearch engine that aggregates results from multiple search engines without tracking users. This deployment features proper SOPS encryption, IP whitelisting, and integration-ready JSON API. Features Privacy-focused: No user tracking or data collection Multi-engine aggregation: Combines results from Google, Bing, DuckDuckGo, Brave, Wikipedia, and more JSON API: RESTful API for programmatic access (perfect for AI integration) Rate limiting with IP whitelisting: Protects against abuse while allowing legitimate usage HTTPS with automatic certificates: Let’s Encrypt via cert-manager SOPS-encrypted secrets: Secure secret management following GitOps best practices Repository Structure ├── apps/ │ └── searxng/ │ └── base/ │ ├── kustomization.yaml │ ├── searxng-namespace.yaml │ ├── searxng-settings.yaml │ ├── searxng-deployment.yaml │ ├── searxng-service.yaml │ ├── searxng-certificate.yaml │ └── searxng-ingress.yaml ├── infrastructure/ │ └── security/ │ └── searxng-secrets/ │ ├── kustomization.yaml │ └── searxng-secret.yaml # SOPS encrypted └── clusters/ └── production/ ├── apps/ │ └── kustomization.yaml # References searxng └── flux-system/ ├── kustomization.yaml # References searxng-secrets └── searxng-secrets.yaml # Flux Kustomization Deployment Steps 1. Create Application Structure Create the application folder structure: ...
Install Flux CLI Install the Flux CLI using the following command: curl -s https://fluxcd.io/install.sh | sudo bash Configure GitHub Repository 1. Create an Empty Repository Go to GitHub and create a new, empty repository. 2. Generate a Personal Access Token Generate a token at GitHub Personal Access Tokens . The token must have the following minimum permissions: Category Permission Metadata Read-only Actions Read and write Administration Read and write Commit statuses Read and write Contents Read and write Dependabot alerts Read and write Dependabot secrets Read and write Deployments Read and write Discussions Read and write Environments Read and write Issues Read and write Merge queues Read and write Pull requests Read and write Repository security advisories Read and write Secret scanning alerts Read and write Secrets Read and write Variables Read and write Webhooks Read and write Workflows Read and write Bootstrap the Cluster 1. Export Required Variables Export your GitHub username, repository name, and token as environment variables: ...
Overview Created a Syncthing pod in Kubernetes cluster managed by FluxCD with dual NFS mounts, SSL certificate via cert-manager, and consolidated LoadBalancer services. Architecture Namespace: syncthing Deployment: Single replica with Recreate strategy Storage: Two NFS persistent volumes SSL: Automatic Let’s Encrypt certificate via cert-manager Load Balancing: Combined TCP/UDP service on single external IP Storage Configuration NFS Mounts # Data mount (Dropbox sync) xxx.xxx.xxx.xxx:/mnt/media/dropbox → /var/syncthing/dropbox # Config mount (Syncthing configuration) xxx.xxx.xxx.xxx:/mnt/media/home/nfs/syncthing → /var/syncthing/config Persistent Volumes syncthing-dropbox-pv: 1Ti capacity for sync data syncthing-config-pv: 1Gi capacity for configuration Both use NFS storage class with ReadWriteMany access mode. ...