Cloud Costs Monitor – Multi-Cloud Cost Management System

This project is a self-hosted, multi-cloud cost management system for AWS, Azure, GCP, and Kubernetes (via a third-party cost attribution platform) that I designed and built from scratch. It runs daily to collect granular cost line items at the service × region × account level, detects spend anomalies, sends configurable alerts, and produces a rich suite of reports in HTML, Excel, and PDF — from executive summaries and rolling-window trend views to per-owner and per-department drilldowns and an annual budget tracker. All data is stored locally in a local database, keeping the system lightweight and self-contained with no external SaaS dependencies.

Key Contributions

• Multi-cloud data collection via AWS Cost Explorer, Azure Cost Management, GCP BigQuery Billing Export, and a Kubernetes namespace cost attribution platform, stored as granular service × region × account × date line items in a local database.
• 9 core report types with exports in HTML, Excel, and PDF: Executive Summary, Rolling Window, Executive Owners, Rolling Owners, Per-Owner, All Owners (batch), Department, All Departments (batch), and Alert Owner.
• Anomaly detection engine that flags spend changes only when both a configurable percentage threshold and a configurable absolute USD threshold are exceeded simultaneously, with a noise floor to suppress false positives from negligible-cost services.
• Budget Tracker showing YTD spend vs annual budget with a pro-rated period budget, an annualised forecast based on daily run rate, a progress bar with a pace marker, and four status badges (ON TRACK / AT RISK / OVER BUDGET / UNBUDGETED).
• Owner-centric YAML model (owners.yaml) mapping cloud account IDs to named owners and departments with annual budgets, email delivery targets, and integrity checks (double-count and over-allocation warnings).
• Efficient historical backfill using date-range API calls rather than day-by-day calls, supporting a gap-fill mode and safe re-runs via UPSERT semantics.
• Alerting via SMTP email and webhooks (Slack, HTTP webhook) with per-owner email delivery that auto-CCs department heads and attaches the HTML report as a MIME attachment.
• Unified Python CLI (cc.py) with five subcommands — run, backfill, report, export, account-map — replacing per-script entry points with a single cross-platform interface.
• Modular code architecture: providers, auth, analyzers, notifiers, cli, export, and budgeting packages with clean separation of concerns.
• 1000+ automated tests across 51 test files: providers (AWS, Azure, GCP, Kubernetes), analyzers, budget calculations, report renderers, CLI commands, Excel exports, and 43 cross-report numerical consistency checks.

Highlights

• All HTML reports are self-contained, fully styled files — emailable directly or opened in any browser with no server required. Excel and PDF exports maintain full section parity with the HTML output. The account-map command additionally produces a Markdown export ready to paste into Confluence, GitHub, or any wiki.
• Dual-threshold detection prevents alert fatigue: a service jumping from $1 to $2 (100 % but negligible USD) does not trigger an alert.
• Azure auto-discovery: if subscriptions: is omitted from config.yaml, the system discovers all subscriptions for a tenant via az account list.
• Azure rate-limit handling: automatic exponential backoff with Retry-After header respect, plus a configurable inter-subscription delay to avoid 429 errors during large backfills.
• Budget over-allocation and account double-count issues surface as visible warning banners in every affected report rather than silently skewing numbers.
• Live validation via cc run validate: SMTP connectivity probe, credential auth tests for all configured providers.
• Interactive module dependency graph (graph.html, 56 modules / 113 import edges) and Mermaid workflow diagrams (workflows.html) for 10 operational workflows.
• The system explicitly does not replace cloud-native billing UIs — it aggregates and reports across providers for a single unified view.

Architecture

The system follows a layered, plugin-style design:

Cloud APIs (AWS / Azure / GCP / K8s)
        │
        ▼
src/auth/*            ← per-provider auth (profile, service principal, OAuth2, instance role)
        │
        ▼
src/providers/*       ← BaseProvider ABC; one implementation per cloud
        │
        ▼
src/database.py       ← database upsert into daily_costs + cost_line_items + k8s_namespace_costs
        │
   ┌────┴────────────────┬──────────────────┐
   ▼                     ▼                  ▼
src/analyzers/*      src/*_report.py    src/budgeting/*
(anomaly detect)     (HTML/Excel/PDF)   (planning tools)
   │                     │
   ▼                     ▼
src/notifiers/*      reports/{html,xlsx,pdf,md}
(email / webhook)
        │
        ▼
src/cli/*             ← unified cc.py: run, backfill, report, export, account-map
src/export/*          ← org map, account map (PDF/MD), owners CSV

Key source modules

Database schema

Three tables with UNIQUE constraints and UPSERT semantics make all operations idempotent. Connection-level performance tuning and three targeted indexes on cost_line_items optimize query performance for large datasets.

daily_costs — aggregated daily total per account (provider, account_id, cost_date, total_cost, currency).

cost_line_items — granular service × region records (provider, account_id, cost_date, service_name, region, cost, currency, plus JSON columns for provider-specific dimensions, tags, and metadata).

k8s_namespace_costs — Kubernetes namespace-level attribution joined with cost_line_items for unified drilldown.

Reports Overview

All report types support --format HTML, --format Excel, --format PDF, and combined flags such as --format HTML,Excel. PDF rendering requires WeasyPrint or wkhtmltopdf.

Anomaly Detection

On each daily run the engine compares every account × service × region trio from yesterday against the prior day. A notable change is flagged only when all of the following conditions hold simultaneously, preventing noise:

1. |Δ%| ≥ percentage_change
2. |Δ USD| ≥ absolute_change
3. Prior cost ≥ min_cost_for_percentage floor — waived for tiny services

Alert direction (increases only, or both increases and decreases) is configurable.

Budget Tracker

Each HTML or Excel report with a Budget Tracker section displays:

Status thresholds are configurable; UNBUDGETED when no budget is configured.

Windows Portable Executable

For deployment on Windows servers without a Python installation, the project compiles to a self-contained cc.exe using PyInstaller. The executable bundles the entire Python 3.11 interpreter, all cloud SDKs (boto3, azure-sdk, GCP client), and all application modules — no Python required on the target machine.

Note: --onedir mode was chosen over --onefile (single-file exe) to avoid AV heuristics triggered by self-extracting packed binaries, and to eliminate the startup overhead of extracting ~250 MB to %TEMP% on every run.