Senior Software Engineer – Data Ingest Engineering(Kafka)
Job Title: Sr Engineer, Software – Data Ingest Engineering
Location: Hyderabad
Commitment: Full-time
Industry: Technology & Engineering
Seniority Level: Senior
Overview
We are looking for a Senior Engineer, Software – Data Ingest Engineering to join a central engineering team responsible for building an Enterprise Unified Ingestion Framework that delivers governed, scalable, and high-performance data to multiple consumers, including applications, business users, and AI-driven solutions.
In this role, you will design and build batch and streaming data ingestion frameworks, cloud-native data pipelines, and scalable data platforms on Azure and Databricks. You will also drive engineering best practices, mentor engineers, and contribute to reusable platform capabilities.
Key Responsibilities
- Design and implement high-throughput, event-driven architectures for near real-time data ingestion into Azure Databricks Lakehouse.
- Build cloud-native data pipelines supporting batch ingestion workloads.
- Configure Unity Catalog by implementing Bronze, Silver, and Gold Medallion Architecture.
- Optimize centralized data governance, fine-grained access control, and end-to-end data lineage across Databricks workspaces.
- Establish standardized DevOps practices for data engineering.
- Optimize data engineering code for scalability, reliability, and performance.
- Identify opportunities to automate data engineering processes and platform operations.
- Partner with domain teams to onboard new data capabilities and use cases.
- Mentor engineers and contribute reusable blueprints and reference implementations.
- Participate in technical leadership and engineering decision-making across global teams.
Required Qualifications
- 7+ years of software engineering experience focused on data infrastructure and backend systems.
- Hands-on experience with the Microsoft Azure Data Platform, including:
- Azure Data Factory (ADF)
- Azure Event Hubs
- Azure Blob Storage
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure DevOps for CI/CD and orchestration
- Production experience with Azure Databricks.
- Strong experience with Delta Lake.
- Hands-on experience with PySpark for distributed data processing and transformation.
- Experience building streaming data pipelines using Kafka or similar streaming platforms.
- Experience implementing Change Data Capture (CDC) and incremental batch processing patterns.
- Strong proficiency in Python.
- Strong proficiency in SQL.
- Experience designing Lakehouse architecture and enterprise data platforms.
- Experience working with cloud-native data engineering tools.
- Strong communication skills with the ability to influence senior stakeholders.
- Ability to lead technical decisions across globally distributed engineering teams.