Data Platform Engineer

Key Responsibilities

• Build, test, and maintain data pipelines and data integration workflows using tools like Spark / PySpark, Airflow, NiFi, Kafka, etc.
• Work with Hive, Iceberg, SQL for data storage, querying, and table formats / lakehouse architectures.
• Integrate batch and streaming data sources, ensuring data quality, consistency, and integrity.
• Optimize performance of ETL / data jobs — tuning Spark jobs, memory, partitions, resource utilization.
• Ensure data security, compliance, and governance (access controls, encryption, compliance standards).
• Work with infrastructure / DevOps teams to deploy, monitor, and scale platform components.
• Troubleshoot production issues, perform root cause analysis, and implement fixes.
• Document processes, technical designs, runbooks, and best practices.
• Mentor junior engineers and contribute to architecture discussions.
• Stay current with emerging data technologies and propose adoption when applicable.