Senior Data Engineer
The IT team is modernizing its data and integration platforms by migrating ETL processes from Workato to Databricks LakeFlow and transitioning application support to a tiered build/run (Tier I/II/III) model.
The Senior Data Engineer will maintain existing data pipelines and Power BI reporting while leading the migration from Workato ETL to Databricks LakeFlow. The role will also involve designing a medallion-architecture data model, managing Epicor Kinetic integration, and establishing governance standards for the data platform.
Experience: 7+ years in data engineering, with 3+ years of hands-on Databricks experience.
Key Responsibilities
BAU Support and Reporting
Maintain and monitor existing data flows from Kinetic (Epicor) and Salesforce through Workato into Databricks.
Troubleshoot pipeline failures, data quality issues, and latency, and resolve root causes.
Maintain and enhance the existing semantic model supporting BAU Power BI reports.
Work with business users to deliver reporting enhancements and new metrics.
Workato to Databricks LakeFlow Migration
Plan the migration by inventorying existing Workato ETL flows, assessing dependencies, and defining a phased cutover with rollback paths.
Build ingestion and transformation pipelines in Databricks LakeFlow using Lakeflow Connect, Declarative Pipelines, and Jobs.
Run parallel validation to reconcile data between existing and new pipelines before decommissioning Workato ETL flows.
Deliver the migration on schedule and report progress and risks to stakeholders.
Data Modeling and Medallion Architecture
Design a comprehensive raw data model that brings data from source systems into Bronze, Silver, and Gold layers.
Model curated Gold-layer datasets to support Power BI and future analytics use cases.
Implement incremental loads, change data capture, schema evolution, and data quality expectations.
Epicor Integration
Design and write Epicor Kinetic Business Activity Queries (BAQs) to expose source tables and views to Databricks, reducing ETL rework whenever a field is added.
Work with the Epicor team ahead of the 2027 Epicor cloud migration, which will remove direct SQL access and route all data extraction through BAQs.
Governance, Standards, and Documentation
Create and document platform and process governance, including standards, policies, naming conventions, and coding practices.
Define access control, data lineage, and cataloguing using Unity Catalog.
Establish CI/CD, version control, and environment promotion for Databricks assets.
Write runbooks and technical documentation to ensure long-term platform support.
Required Qualifications
7+ years of experience in data engineering, including 3+ years building production pipelines on Databricks.
Strong Python and advanced SQL skills, with practical experience in Delta Lake and medallion architecture.
Hands-on production experience with dbt. This is a mandatory requirement.
Hands-on experience with Databricks LakeFlow, Delta Live Tables / Declarative Pipelines, or Databricks Workflows.
Experience with Unity Catalog for governance, permissions, and data lineage.
Experience building semantic models for Power BI, including star schema, DAX basics, and performance tuning.
Hands-on experience with Epicor Kinetic data structures and Business Activity Queries (BAQs), along with Salesforce or a comparable CRM as a data source.
Proven track record of leading an ETL or platform migration end to end.
Experience with Git-based workflows and CI/CD using Azure DevOps, GitHub Actions, or Databricks Asset Bundles.
Experience with Microsoft Azure. The data platform runs on Azure; AWS-only experience is not suitable.
Strong written documentation and stakeholder communication skills.
Preferred Qualifications
Experience with Workato or another iPaaS tool.
Databricks Certified Data Engineer Professional or Associate certification.