Databricks Data Engineer

Guidehouse is hiring a Databricks Data Engineer for its AI & Data team to support client projects involving large-scale data pipelines, platform modernization, analytics, and AI solutions. The role designs, builds, and maintains scalable data solutions and collaborates with clients and cross-functional teams to deliver reliable, secure, and documented data engineering work.

Responsibilities

Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, Delta Lake, and related cloud-native data engineering tools

Develop and support batch and streaming data workflows, including ingestion, transformation, validation, and publishing of curated data products

Write, optimize, and maintain Python, PySpark, and SQL code for data processing, orchestration, and performance tuning

Work with large-scale datasets using Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage services

Troubleshoot and resolve data pipeline failures, performance issues, data quality issues, and workflow bottlenecks

Translate business and technical requirements into data engineering designs, processing scripts, and job orchestration workflows

Perform data validation, quality checks, code reviews, and issue resolution to support accurate and reliable data products

Collaborate with cross-functional teams including solution architects, data scientists, analysts, DevOps engineers, and client stakeholders

Communicate technical concepts, delivery impacts, and data engineering considerations to technical and non-technical audiences

Document pipelines, datasets, data models, workflows, and engineering decisions to support maintainability, transparency, and reuse

Follow data governance, security, lineage, and compliance standards within the Databricks platform

Preferred

2+ years of hands-on experience with the Databricks platform

Experience using Unity Catalog for data governance, schema design, data modeling standards, access controls, lineage, and secure management of enterprise data assets

Experience designing and building standard medallion data architectures for scalable ingestion, transformation, quality validation, and curated data delivery

Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification

Experience with CI/CD tools and practices for notebooks, workflows, orchestration jobs, and infrastructure deployment

Experience with cloud platforms such as Azure, AWS, or GCP

Experience working in project-based or consulting delivery environments

Familiarity with streaming frameworks, orchestration tools, Terraform, or related data platform administration capabilities

2+ years of hands-on experience with the Databricks platform.

Databricks Platform

Schema Design

Data Modeling Standards

Access Controls

Enterprise Data Security

Medallion Architecture

Scalable Data Ingestion

Curated Data Delivery

Databricks Data Engineer Associate Certification

Databricks Data Engineer Professional Certification

CI/CD

Notebook Development

Workflow Management

Job Orchestration

Infrastructure Deployment

Azure

AWS

GCP

Project-Based Delivery

Consulting Delivery

Streaming Frameworks

Orchestration Tools

Terraform

Data Platform Administration

Benefits

This position offers virtual work flexibility within the United States.

A flexible benefits package

Company

AI-led advisory, technology, and managed services for enterprises

Mclean, Virginia, USA

https://guidehouse.com/

Founded in 2018

10001+ employees

Qualifications

Bachelor's degree in computer science, engineering, mathematics, statistics, or another relevant field

3-8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation

Strong experience with Python, PySpark, and SQL for data transformation, pipeline development, and data processing

Experience with Databricks, Spark, Delta Lake, Unity Catalog, or similar cloud-native data platforms

Experience developing batch or streaming data pipelines, ETL/ELT workflows, and reusable data assets

Experience with data modeling, data warehousing, data quality validation, and large-scale data processing concepts

Ability to troubleshoot technical issues, communicate engineering recommendations clearly, and work effectively in team-based delivery environments

Experience applying data governance, access control, lineage, and security practices within cloud data platforms

2+ years of hands-on experience with the Databricks platform

Experience using Unity Catalog for data governance, schema design, data modeling standards, access controls, lineage, and secure management of enterprise data assets

Experience designing and building standard medallion data architectures for scalable ingestion, transformation, quality validation, and curated data delivery

Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification

Experience with CI/CD tools and practices for notebooks, workflows, orchestration jobs, and infrastructure deployment

Experience with cloud platforms such as Azure, AWS, or GCP

Experience working in project-based or consulting delivery environments

Familiarity with streaming frameworks, orchestration tools, Terraform, or related data platform administration capabilities

3-8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation.

Required skills

Python, PySpark, SQL, Databricks, Apache Spark, Delta Lake, Unity Catalog, Data Pipeline Development, ETL/ELT, Data Modeling, Data Governance, Azure

Preferred skills

2+ years of hands-on experience with the Databricks platform, Experience with cloud platforms such as Azure, AWS, or GCP

Experience

3-8 years