Turning Legacy Data Infrastructure into a $50K Monthly Advantage

The job orchestration system running on Airflow 1.x within GCP Cloud Composer had reached end-of-life, creating security and operational risks. The environment had been configured manually, making maintenance risky and updates difficult. We built parallel Airflow 2.x environments as code, developed a comprehensive DAG test suite to validate all job types and custom operators, and migrated pipelines in controlled batches. The entire infrastructure was transitioned to Infrastructure as Code using Terraform, eliminating technical debt and enabling repeatable deployments and centralized management.

The legacy .NET monolithic application for processing member data couldn't scale to handle massive CSV files. We designed and built a new Apache Spark pipeline on Amazon EMR that leverages distributed processing across a cluster. The solution uses a staging database pattern to protect production systems from write load spikes, handles both large batch files and high-volume micro-batches, and provides automated reporting via email and Slack. The architecture was designed for horizontal scalability, allowing the client to increase processing capacity by scaling the EMR cluster rather than rewriting code.

AWS: RDS, EMR, EC2, S3, CloudWatch, Route 53

GCP: Cloud Composer, BigQuery, Cloud Build

Data Processing: Apache Spark, Airflow 2.x, MySQL, MSSQL

Infrastructure as Code: Terraform, YAML, Jinja

Monitoring: Percona PMM, CloudWatch

Data-Driven Decisions: Every sizing and consolidation decision was based on observed performance data, not estimates.

Parallel Environments: New environments were built alongside existing ones, enabling testing without production risk.

Phased Migration: Workloads were migrated in controlled batches with extended monitoring.

Transparent Cutover: Where possible, migrations used DNS changes rather than application code changes.

Zero Downtime: All migrations were executed without service interruption.