ANIRBAN SANYAL

Architected and deployed an end-to-end Dataflow pipeline (Apache Beam / Python) that ingested mainframe files into Bigtable, achieving sub-10ms API-based data retrieval for a US banking system processing millions of daily transactions.

Implemented robust idempotency checks within Dataflow pipelines, eliminating duplicate records during re-runs and ensuring 100% data consistency and integrity across all load cycles.

Developed a comprehensive metadata tracking framework (ABC Framework) in BigQuery, capturing pipeline-level record counts, timestamps, and status flags to enable full data lineage and operational observability.

Engineered structured error capture mechanisms into BigQuery, routing failed records with detailed context to facilitate rapid root-cause analysis and significantly reduce incident resolution time.

Delivered an event-driven orchestration layer utilizing Cloud Functions and Cloud Composer (Airflow) to automatically trigger Dataflow jobs, enabling near real-time processing and reducing manual intervention.

Managed and optimized large-scale analytical datasets in BigQuery and Cloud Storage, supporting production workloads processing tens of millions of records daily across multiple business domains.

Resolved critical data incidents through deep SQL analysis and root-cause investigation, consistently maintaining SLA compliance and minimizing disruptions to customer-facing analytics and reporting pipelines.

Maintained and enhanced CI/CD pipelines using GitHub Actions and Cloud Build, enabling zero-downtime deployments and supporting comprehensive data investigations for funnel and customer journey analytics.

Designed and implemented scalable batch and streaming data pipelines using Pub/Sub, Python, SQL, and Cloud Composer, ensuring reliable and performant data ingestion across enterprise workloads.

Developed robust data ingestion, transformation, and migration frameworks using Oracle and Oracle Data Integrator (ODI), ensuring high data accuracy and seamless system transitions across upgrades and consolidations.

Collaborated closely with business stakeholders to translate complex requirements into scalable technical pipeline designs and resolved data quality issues through structured root-cause analysis, improving system reliability.