Senior Software Engineer – Data Ingest Engineering(Kafka)
Hyderabad Full-time Technology & Engineering Senior
What You’ll Do
- Design and implement high-throughput, event-driven architectures to power near real-time data pipelines for ingesting into Azure + Databricks Lakehouse
- Build cloud-native data pipelines to support batch ingest workloads
- Configure Unity Catalog by designing Bronze/Silver/Gold medallion architecture and optimize for centralized data governance, fine-grained access control, and end-to-end lineage tracking across Databricks workspaces
- Drive engineering excellence by setting standardized DevOps practices and optimizing data engineering code for performance
- Identify and design solutions to potential data processes that can be improved with automation
- Partner with domain teams to onboard new capability use cases, mentor engineers, and contribute to reusable blueprints and reference implementations.
What You’ll Bring
- 7+ years of software engineering experience focused on data infrastructure and backend systems.
- Hands-on experience with the Microsoft Azure data platform — Azure Data Factory, Event Hubs, Blob Storage, ADLS Gen2, and Azure DevOps for orchestration and CI/CD
- Production experience with Azure Databricks and Delta Lake
- Production experience with PySpark for distributed data processing and transformation.
- Streaming frameworks (Kafka or comparable) for real-time data paths including CDC and incremental batch patterns.
- Strong communication skills with the ability to influence senior stakeholders and lead technical decisions across global teams.
Must Have Skills
- Data Platform Architecture (Lakehouse, Delta Lake, Azure Databricks)
- Kafka / Streaming Frameworks
- Strong SQL and Python Proficiency
- Experience with Cloud Native Tools
Nice-to-Have
- Building data infrastructure for AI use cases
- Prior experience in large-enterprise data integration or platform engineering environments
- Prior experience in enterprise platform migrations