Senior Software Engineer – Data Ingest Engineering(ETL/ELT)
Job Title: Senior Engineer, Software – Data Ingest Engineering
Location: Hyderabad
Commitment: Full-time
Industry: Technology & Engineering
Seniority Level: Senior
Overview
We are looking for a Senior Engineer, Software – Data Ingest Engineering to design, build, and enhance an enterprise-scale unified data ingestion framework that delivers data securely, efficiently, and at scale. The role focuses on developing cloud-native batch and streaming data pipelines, enabling governed data access, and supporting modern analytics and AI-driven use cases through Azure and Databricks technologies.
As a hands-on engineer, you will contribute to the design and implementation of scalable data infrastructure while collaborating with cross-functional teams to build reusable frameworks and engineering best practices.
Key Responsibilities
- Design and implement high-throughput, event-driven architectures for near real-time data ingestion into Azure and Databricks Lakehouse.
- Build cloud-native data pipelines to support batch ingestion workloads.
- Configure Unity Catalog using Bronze, Silver, and Gold Medallion architecture.
- Optimize centralized data governance, fine-grained access control, and end-to-end data lineage across Databricks workspaces.
- Drive engineering excellence by defining standardized DevOps practices.
- Optimize data engineering code for scalability and performance.
- Identify opportunities to automate and improve existing data engineering processes.
- Partner with domain teams to onboard new data ingestion capabilities.
- Mentor engineers and contribute reusable blueprints and reference implementations.
- Support enterprise-scale data infrastructure for batch and streaming workloads.
Required Qualifications
- 7+ years of software engineering experience focused on data infrastructure and backend systems.
- Hands-on experience with Microsoft Azure Data Platform, including:
- Azure Data Factory (ADF)
- Event Hubs
- Blob Storage
- Azure Data Lake Storage (ADLS Gen2)
- Azure DevOps for orchestration and CI/CD
- Production experience with Azure Databricks.
- Strong experience with Delta Lake.
- Production experience using PySpark for distributed data processing and transformation.
- Experience with streaming platforms such as Kafka or comparable technologies.
- Experience implementing Change Data Capture (CDC) and incremental batch processing.
- Strong proficiency in SQL.
- Strong proficiency in Python.
- Experience designing Lakehouse-based data platform architectures.
- Experience with cloud-native data engineering tools.
- Knowledge of ETL and ELT data processing frameworks.
- Strong communication skills with the ability to influence senior stakeholders.
- Ability to lead technical discussions across globally distributed teams.