Senior Software Engineer – Data Ingest Engineering(Kafka)

Job Title: Sr Engineer, Software – Data Ingest Engineering

Location: Hyderabad

Commitment: Full-time

Industry: Technology & Engineering

Seniority Level: Senior

Overview

We are looking for a Senior Engineer, Software – Data Ingest Engineering to join a central engineering team responsible for building an Enterprise Unified Ingestion Framework that delivers governed, scalable, and high-performance data to multiple consumers, including applications, business users, and AI-driven solutions.

In this role, you will design and build batch and streaming data ingestion frameworks, cloud-native data pipelines, and scalable data platforms on Azure and Databricks. You will also drive engineering best practices, mentor engineers, and contribute to reusable platform capabilities.

Key Responsibilities

• Design and implement high-throughput, event-driven architectures for near real-time data ingestion into Azure Databricks Lakehouse.
• Build cloud-native data pipelines supporting batch ingestion workloads.
• Configure Unity Catalog by implementing Bronze, Silver, and Gold Medallion Architecture.
• Optimize centralized data governance, fine-grained access control, and end-to-end data lineage across Databricks workspaces.
• Establish standardized DevOps practices for data engineering.
• Optimize data engineering code for scalability, reliability, and performance.
• Identify opportunities to automate data engineering processes and platform operations.
• Partner with domain teams to onboard new data capabilities and use cases.
• Mentor engineers and contribute reusable blueprints and reference implementations.
• Participate in technical leadership and engineering decision-making across global teams.

Required Qualifications

• 7+ years of software engineering experience focused on data infrastructure and backend systems.
• Hands-on experience with the Microsoft Azure Data Platform, including:
  • Azure Data Factory (ADF)
  • Azure Event Hubs
  • Azure Blob Storage
  • Azure Data Lake Storage Gen2 (ADLS Gen2)
  • Azure DevOps for CI/CD and orchestration
• Production experience with Azure Databricks.
• Strong experience with Delta Lake.
• Hands-on experience with PySpark for distributed data processing and transformation.
• Experience building streaming data pipelines using Kafka or similar streaming platforms.
• Experience implementing Change Data Capture (CDC) and incremental batch processing patterns.
• Strong proficiency in Python.
• Strong proficiency in SQL.
• Experience designing Lakehouse architecture and enterprise data platforms.
• Experience working with cloud-native data engineering tools.
• Strong communication skills with the ability to influence senior stakeholders.
• Ability to lead technical decisions across globally distributed engineering teams.