Big Data Platform Engineer/Data Platform Analyst (Azure, Spark, Airflow, Kubernetes)

🌍 Remote, USA 🎯 Full-time 🕐 Posted Recently

Job Description

Dice is the leading career destination for tech experts at every stage of their careers. Our client, ERPA, is seeking the following. Apply via Dice today!

Azure
Airflow
Spark
Hive
Kubernetes

BDP Platform Engineer — Contractor Lead (Onsite)

Client: US Bank

Locations: Minneapolis, Atlanta, or San Francisco

Duration: 12-24 months

Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite–offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.

Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
Onsite–offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.

Experience: 4–8+ years in platform/data engineering with production ownership.
Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
Languages: Proficient in Python, Scala, and Java.
DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.