Job Description
Note: The job is a remote job and is open to candidates in USA. CloudTech Innovations is seeking a highly experienced Solution Architect with a strong background in building and scaling cloud-native data platforms. This role focuses on Databricks and involves collaborating with cross-functional teams to design and implement well-architected data solutions across enterprise environments.
- Responsibilities
- Design and lead implementation of end-to-end Databricks Lakehouse Platforms using Delta Lake, Delta Live Tables, and MLflow
- Architect Medallion Architecture (Bronze/Silver/Gold) for structured, semi-structured, and streaming workloads
- Implement governed Lakehouse patterns using Unity Catalog for access control, lineage, data classification, and secure sharing
- Build scalable ETL/ELT pipelines using Databricks Notebooks, Workflows, SQL Warehouses, and Spark-based transformations
- Develop real-time streaming pipelines with Auto Loader, Structured Streaming, and event-driven platforms (Kafka, Kinesis, Pub/Sub)
- Integrate Databricks with cloud-native services such as AWS Glue, Azure Data Factory, and GCP Dataform
- Define distributed integration patterns using REST APIs , microservices, and event-driven architectures
- Enforce data governance, RBAC/ABAC, encryption, secret management , and compliance controls
- Optimize Delta Lake tables, Spark workloads, and cluster configurations using Photon and autoscaling patterns
- Drive cloud cost optimization across storage, compute, and workflow orchestration
- Participate in architecture reviews, set standards, and support engineering teams throughout execution
- Stay current on Databricks capabilities including Unity Catalog updates, Lakehouse Federation, serverless compute, and AI/ML features
- Skills
- Bachelor's or Master's degree in Computer Science, Data Engineering, or related field
- 10+ years of experience in enterprise software, cloud architecture, or data engineering roles
- Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and Lakehouse platform design
- Experience implementing and administering Unity Catalog for governance, lineage, and fine-grained access control
- Experience designing Medallion Architecture for analytics and engineering workloads
- Hands-on experience with cloud platforms such as AWS, Azure, or GCP, including storage, compute, and networking services
- Experience with streaming technologies such as Kafka, Kinesis, or Pub/Sub
- Strong understanding of data modeling, workflow orchestration (Airflow, Databricks Workflows, dbt), and pipeline automation
- Familiarity with Scala-based Spark workloads in addition to PySpark and SQL pipelines
- Skilled in performance tuning, Spark optimization, cluster policies, and cloud cost management
- Excellent communication skills for technical leadership and stakeholder collaboration
- Certifications in Databricks, AWS/GCP/Azure Solution Architecture, or TOGAF are a plus
- Company Overview
- Comprehensive Staffing Solutions for Specialized Engineering Roles: Software Development Engineering: We offer staffing solutions with skilled software engineers proficient in various programming languages and technologies. It was founded in 2023, and is headquartered in Dallas, TX, US, with a workforce of 51-200 employees. Its website is https://cloudtech-innovations.com.
Apply tot his job
Apply To this Job