Consultant, Data Engineer

Infinitive is a data and AI consultancy that hires a Consultant, Data Engineer to build and maintain Databricks data pipelines for its clients. The role writes production Python, PySpark and SQL code, supports data quality and pipeline operations, and works with client teams to clarify requirements and explain technical work.

Responsibilities

Build and maintain batch and incremental ETL pipelines in Databricks using Python, PySpark and Spark SQL

Ingest data from relational databases, APIs, flat files and cloud storage (Amazon S3, Azure Data Lake Storage) into Delta Lake tables

Implement bronze, silver and gold layers following medallion architecture patterns

Write and tune SQL for transformations, data validation and reporting tables

Add data quality checks, handle schema changes and troubleshoot failed jobs

Schedule and monitor workloads with Databricks Workflows, working within Unity Catalog permissions

Tune Spark jobs for performance and cost, including partitioning, file sizing and cluster configuration

Use Git for version control, take part in code reviews and contribute to CI/CD pipelines

Document pipelines, source-to-target mappings and runbooks so client teams can run what you build

Join client meetings to clarify requirements, report status and walk through your work

Preferred

Databricks Certified Data Engineer Associate certification

Experience with Delta Lake features such as MERGE and time travel, or with Lakeflow Declarative Pipelines (formerly Delta Live Tables)

Exposure to Unity Catalog, dbt, Apache Airflow or Azure Data Factory

Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps

Prior consulting or client-facing experience

Background in financial services, healthcare or media data

Databricks Certified Data Engineer Associate

Delta Lake

MERGE Operations

Time Travel

Lakeflow Declarative Pipelines

Unity Catalog

dbt

Apache Airflow

Azure Data Factory

Terraform

CI/CD

GitHub Actions

Azure DevOps

Consulting

Financial Services Data

Healthcare Data

Media Data

Client-Facing Communication

Benefits

This is a hybrid role and requires work on-site with our clients

Company

Data and AI consultancy for enterprise digital transformation

Ashburn, Virginia, USA

http://www.infinitive.com/

Founded in 2003

51-200 employees

Qualifications

1 to 3 years of professional experience in data engineering, software engineering or a related technical role

Bachelor's degree in computer science, information systems, engineering, mathematics or a related field, or equivalent experience

Hands-on experience with Databricks on a professional or substantial academic project

Working proficiency in Python for data processing

Experience building transformations with Apache Spark and PySpark (DataFrames and Spark SQL)

Strong SQL, including joins, window functions, CTEs and aggregations

Understanding of ETL concepts such as incremental loads, slowly changing dimensions, dimensional modeling and data quality checks

Familiarity with at least one cloud platform (AWS, Azure or Google Cloud)

Clear written and verbal communication, and comfort explaining technical work to non-technical stakeholders

Authorization to work in the United States

Databricks Certified Data Engineer Associate certification

Experience with Delta Lake features such as MERGE and time travel, or with Lakeflow Declarative Pipelines (formerly Delta Live Tables)

Exposure to Unity Catalog, dbt, Apache Airflow or Azure Data Factory

Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps

Prior consulting or client-facing experience

Background in financial services, healthcare or media data

Required skills

Databricks, Python, Apache Spark and PySpark, SQL, ETL, Dimensional Modeling, Data Quality, AWS, Azure, Google Cloud, Databricks Certified Data Engineer Associate, Delta Lake

Preferred skills

Databricks Certified Data Engineer Associate certification, Prior consulting or client-facing experience, Background in financial services, healthcare or media data

Experience

1-3 years