Job Description
Title: Data Engineer/Architect - Cloudera NiFi
Location: Remote
Duration: 6 Months Contract
Key Responsibilities
- NiFi Platform Assessment & Optimization
- Review existing Cloudera NiFi flows and architectures for efficiency, scalability, and maintainability.
- Assess current use cases to ensure NiFi is being applied appropriately and effectively.
- Identify opportunities to simplify flow designs, reduce operational overhead, and improve performance.
- Provide recommendations on best practices, standards, and reusable patterns.
- Data Integration & Pipeline Support
- Support and enhance ETL and ELT pipelines built on NiFi.
- Integrate data from diverse sources including databases, APIs, filesystems, streaming platforms, and cloud storage.
- Troubleshoot data flow failures, latency issues, and throughput constraints.
- Implement robust error handling, monitoring, and recovery mechanisms.
- Python & Custom Development
- Use Python for transformations, scripting, automation, and custom processor logic (where applicable).
- Support integration between NiFi and external systems using custom scripts or utilities.
- Governance, Security & Operations
- Apply best practices for secure data movement, authentication, authorization, and encryption.
- Assist with flow versioning, deployment processes, and operational runbooks.
- Collaborate with internal teams to ensure platform changes align with operational and compliance requirements.
- Advisory & Collaboration
- Act as a technical advisor to internal data and platform teams.
- Clearly document findings, recommendations, and implemented improvements.
- Provide knowledge transfer and guidance to ensure long‑term sustainability.
- Required Experience & Skills
- Proven, hands‑on experience with Cloudera NiFi in production environments.
- Strong knowledge of ETL/ELT concepts and modern data pipeline architectures.
- Proficiency in Python for data processing, scripting, and automation.
- Experience working with Linux‑based systems and distributed environments.
- Ability to analyze existing implementations and deliver actionable optimization recommendations.
- Preferred Experience
- Experience within the Cloudera ecosystem (e.g., Kafka, HDFS, Hive, Iceberg).
- Familiarity with CI/CD practices for data pipelines.
- Experience with monitoring and tuning distributed data platforms.
- Exposure to cloud and hybrid data architectures.
- Engagement Expectations
- Deliverables‑focused, hands‑on consulting role.
- Ability to work independently while collaborating with internal stakeholders.
- Clear written and verbal communication of technical findings and recommendations.
Apply tot his job
Apply To this Job