Job Description
- --- Project requirements mandate role open only for US Citizens. IRS MBI Clearance a plus/ Active Secret or Top Secret a Plus. All candidates will have to go through Clearance process before being able to start on the project.--(No exceptions to this requirement) Job Description • Infobahn Solutions is hiring Databricks Data Engineering professionals in the Washington DC Metro Area for a US Government Federal Project with the Department of Treasury . • The Data Engineers will be part of a Data Migration & Conversion Team on a large DataLake being implemented on AWS Gov Cloud . • Data will be migrated from on premise Main Frame /Legacy database systems using Informatica PowerCenter to the AWS Landing Zone on S3. • Further conversion will be done using Databricks (PySpark) in AWS. • The Data Engineer should have prior Data Migration experience and understand all the intricacies required of developing data integration routines for moving data from multiple source systems to a new target system with a different data model. • The Data Engineer should have experience in converting Oracle PL/SQL and/or Greenplum code to Databricks. • Must have experience - Experience with Data Migrations and Conversion using Databricks . • Experience of using Databricks on AWS and managing a Databricks production system is critical and a must have for the project. What you’ll be doing: • Databricks Environment Setup: Configure and maintain Databricks clusters, ensuring optimal performance and scalability for big data processing and analytics. • ETL (Extract, Transform, Load): Design and implement ETL processes using Databricks notebooks or jobs to process and transform raw data into a usable format for analysis. • Data Lake Integration: Work with data lakes and data storage systems to efficiently manage and access large datasets within the Databricks environment. • Data Processing and Analysis: Develop and optimize Spark jobs for data processing, analysis, and machine learning tasks using Databricks notebooks. • Collaboration: Collaborate with data scientists, data engineers, and other stakeholders to understand business requirements and implement solutions. • Performance Tuning: Identify and address performance bottlenecks in Databricks jobs and clusters to optimize data processing speed and resource utilization. • Security and Compliance: Implement and enforce security measures to protect sensitive data within the Databricks environment, ensuring compliance with relevant regulations. • Documentation: Maintain documentation for Databricks workflows, configurations, and best practices to facilitate knowledge sharing and team collaboration. Skills: • Apache Spark: Strong expertise in Apache Spark, which is the underlying distributed computing engine in Databricks. • Databricks Platform: In-depth knowledge of the Databricks platform, including its features, architecture, and administration. • Programming Languages: Proficiency in languages such as Python or Scala for developing Spark applications within Databricks. • SQL: Strong SQL skills for data manipulation, querying, and analysis within Databricks notebooks. • ETL Tools: Experience with ETL tools and frameworks for efficient data processing and transformation. • Data Lake and Storage: Familiarity with data lakes and storage systems, such as Delta Lake, AWS S3, or Azure Data Lake Storage. • Collaboration and Communication: Effective communication and collaboration skills to work with cross-functional teams and stakeholders. • Problem Solving: Strong problem-solving skills to troubleshoot issues and optimize Databricks workflows. • Version Control: Experience with version control systems (e.g., Git) for managing and tracking changes to Databricks notebooks and code. Role Requirements: • Bachelor/Master’s degree in computer science, Engineering, or related field • 7-8 plus years of development experience on ETL tools (4+ years of Databricks is a must have) • 5+ years of experience as a Databricks Engineer or similar role. • Strong expertise in Apache Spark and hands-on experience with Databricks. • More than 7 years of experience performing data reconciliation, data validation, ETL testing, deploying ETL packages and automating ETL jobs, developing reconciliation reports. • Working knowledge of message-oriented middleware/streaming data technologies such as Kafka, Confluent • Proficiency in programming languages such as Python or Scala for developing Spark applications. • Solid understanding of ETL processes and data modeling concepts. • Experience with data lakes and storage systems, such as Delta Lake, AWS S3, or Azure Data Lake Storage. • Strong SQL skills for data manipulation and analysis. • Good experience in shell scripting, AutoSys • Strong Data Modeling Skills • Strong analytical skills applied to business software solutions maintenance and/or development • Must be able to work with a team to write code, review code, and work on system operations. • Past project experience with Data Conversion and Data Migration • Communicate analysis, results and ideas to key decision makers including business and technical stakeholders. • Experience in developing and deploying data ingestion, processing, and distribution systems with AWS technologies • Experience with using AWS datastores, including RDS Postgres, S3, or DynamoDB • Dev-ops experience using GIT, developing, deploying code to production • Proficient in using AWS Cloud Services for Data Engineering tasks • Proficient in programming in Python/shell or other scripting languages for the purpose of data movement • Eligible for a US Government issued IRS MBI (candidates with active IRS MBIs will be preferred) • Databricks industry certifications - Associate / Professional Level Preferred Qualifications • Cloud Data Migration and Conversion projects • Experience on AWS Job Types: Full-time, Contract Pay: $90,000.00 - $130,000.00 per year Benefits: • Dental insurance • Flexible schedule • Health insurance • Life insurance • Paid time off • Vision insurance Education: • Bachelor's (Preferred) License/Certification: • Databricks Certified Data Engineer Professional (Required) Security clearance: • Secret (Preferred) Work Location: Remote Apply tot his job