← Back to jobsJob ID col-cmwpde53y0ayid8ha6yhh8x8f

Senior Data Engineer - Databricks Job DetailsPosted 1 day ago

Senior Data Engineer - DatabricksNew

Full-time
On-site
5–8 years
$89K – $166K/yr
Posted 1 day ago

Capgemini is hiring a Senior Data Engineer - Databricks to design, develop, and optimize scalable data solutions supporting analytics, machine learning, and generative AI. The role builds modern lakehouse architectures and high-performance data pipelines in collaboration with data architects, data scientists, business stakeholders, and application teams.

Posted
1 day ago
Source
careers.capgemini.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader
  • Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources
  • Implement data quality checks, monitoring, and pipeline optimization to ensure reliability and performance
  • Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers)
  • Develop and maintain Delta Lake tables while ensuring optimal query performance
  • Perform performance tuning using Databricks best practices, (e.g. Photon Engine,Liquid Clustering, Partitioning)
  • Configure and administer Unity Catalog to manage data governance across the enterprise
  • Implement catalog, schema, table, row-level, and column-level security controls
  • Enable data lineage, auditing, and compliance requirements across the data platform
  • Develop and maintain workflows using Databricks Workflows
  • Manage deployment processes using Databricks Asset Bundles (DABs)
  • Implement CI/CD pipelines leveraging GitHub, GitLab, and DevOps best practices
  • Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies
  • Prepare and curate datasets for machine learning and generative AI applications
  • Support feature engineering and feature store implementations
  • Enable vector search and retrieval capabilities to support large language model (LLM) solutions and AI-driven use cases
  • Collaborate with Data Science and AI teams to operationalize ML workloads

Qualifications

  • 5-8 years of experience in data engineering and cloud-based data platforms
  • The ideal candidate will have strong hands-on expertise in the Databricks ecosystem, including PySpark, Delta Lake, Unity Catalog, Structured Streaming, and Databricks Workflows
  • 5-8 years of experience in Data Engineering
  • Strong proficiency in **Python (PySpark)** and **SQL**
  • Experience designing and supporting large-scale enterprise data platforms
  • Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns
  • Experience implementing CI/CD pipelines, version control, and automated testing practices
  • Hands-on experience with **Git, GitHub/GitLab, Terraform, and pytest**
  • Experience working with at least one major cloud platform: (Azure, GCP, AWS)

Required skills

Auto LoaderAzure, GCP, or AWSDatabricks Lakehouse PlatformDatabricks WorkflowsDelta LakeGit, GitHub, and GitLabPython (PySpark)SQLStructured StreamingTerraformUnity Catalogpytest

Full job description

Capgemini is hiring a Senior Data Engineer - Databricks to design, develop, and optimize scalable data solutions supporting analytics, machine learning, and generative AI. The role builds modern lakehouse architectures and high-performance data pipelines in collaboration with data architects, data scientists, business stakeholders, and application teams.

Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader
  • Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources
  • Implement data quality checks, monitoring, and pipeline optimization to ensure reliability and performance
  • Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers)
  • Develop and maintain Delta Lake tables while ensuring optimal query performance
  • Perform performance tuning using Databricks best practices, (e.g. Photon Engine,Liquid Clustering, Partitioning)
  • Configure and administer Unity Catalog to manage data governance across the enterprise
  • Implement catalog, schema, table, row-level, and column-level security controls
  • Enable data lineage, auditing, and compliance requirements across the data platform
  • Develop and maintain workflows using Databricks Workflows
  • Manage deployment processes using Databricks Asset Bundles (DABs)
  • Implement CI/CD pipelines leveraging GitHub, GitLab, and DevOps best practices
  • Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies
  • Prepare and curate datasets for machine learning and generative AI applications
  • Support feature engineering and feature store implementations
  • Enable vector search and retrieval capabilities to support large language model (LLM) solutions and AI-driven use cases
  • Collaborate with Data Science and AI teams to operationalize ML workloads

Benefits

  • For regular, full-time employees in the U.S. and Canada, subject to local policy and eligibility: paid time off based on employee grade (A-F): 12-25 vacation days, depending on grade; company-paid holidays; personal days; sick leave.
  • For regular, full-time employees in the U.S. and Canada, subject to local policy and eligibility: medical, dental, and vision coverage (or provincial healthcare coordination in Canada).
  • For regular, full-time employees in the U.S. and Canada, subject to local policy and eligibility: retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada).
  • For regular, full-time employees in the U.S. and Canada, subject to local policy and eligibility: life and disability insurance.
  • For regular, full-time employees in the U.S. and Canada, subject to local policy and eligibility: employee assistance programs.
  • This role may be eligible for additional compensation such as variable incentives, bonuses, or commissions, depending on the position and applicable laws.

Company

  • Consulting, technology services, and digital transformation.
  • Paris, Ile-de-France, FRA
  • https://www.capgemini.com/
  • Founded in 1967
  • 10001+ employees

Qualifications

  • 5-8 years of experience in data engineering and cloud-based data platforms
  • The ideal candidate will have strong hands-on expertise in the Databricks ecosystem, including PySpark, Delta Lake, Unity Catalog, Structured Streaming, and Databricks Workflows
  • 5-8 years of experience in Data Engineering
  • Strong proficiency in **Python (PySpark)** and **SQL**
  • Databricks Lakehouse Platform
  • Delta Lake
  • Unity Catalog
  • Databricks Workflows
  • Structured Streaming
  • Auto Loader
  • Experience designing and supporting large-scale enterprise data platforms
  • Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns
  • Experience implementing CI/CD pipelines, version control, and automated testing practices
  • Hands-on experience with **Git, GitHub/GitLab, Terraform, and pytest**
  • Experience working with at least one major cloud platform: (Azure, GCP, AWS)
  • 5-8 years of experience in data engineering and cloud-based data platforms.
  • 5-8 years of experience in Data Engineering.

Required skills

  • Auto Loader, Azure, GCP, or AWS, Databricks Lakehouse Platform, Databricks Workflows, Delta Lake, Git, GitHub, and GitLab, Python (PySpark), SQL, Structured Streaming, Terraform, Unity Catalog, pytest

Experience

  • 5-8 years

Job details

Employment type

Full-time

Seniority level

5–8 years

Date posted

Nov 20, 2026

Work mode

On-site

Job source

careers.capgemini.com

Company

Senior Data Engineer - Databricks Job Details

Information Technology & AI

Consulting, technology services, and digital transformation.

  • Troy, MI
  • Information Technology & AI
  • 10001+ employees
  • Founded 1967
  • capgemini.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Data Engineer jobs in Troy

Diverse Lynx

1d ago

New

Data Engineer || Quincy, MA

Full-timeOn-siteQuincy, MACompetitive

Diverse Lynx is recruiting a Data Engineer for a confidential client. The role focuses on building and developing data ingestion, streaming, and transformation solutions using Azure Databricks, as well as supporting data warehouse and lakehouse…

Why similarSame roleAuto LoaderDelta Lake

Capgemini

1d ago

New

Senior Data Engineer - Databricks

Full-timeOn-siteNew York, NY$84K – $156K/yr

Capgemini is seeking a Senior Databricks Data Engineer to design, build, and optimize scalable data platforms and pipelines in cloud-based environments.

Why similarSame roleDelta LakeSQL

Dice

1d ago

New

Senior Data Engineer - Databricks

Full-timeOn-siteNew York, NY$84K – $156K/yr

Capgemini is seeking a Senior Databricks Data Engineer to design, build, and optimize scalable cloud-based data platforms and pipelines.

Why similarSame roleDelta LakeSQL

Get new Data Engineer jobs in Troy by email

Create job alert

Senior Data Engineer - Databricks

Senior Data Engineer - Databricks Job Details · $89K – $166K/yr

Apply on company site