← Back to jobsJob ID col-cmwp6g4p206q8d8haezmhacjl

NTT DATAPosted today

Senior Data Engineering - Support EngineerNew

Full-time
Remote
7+ years
$92K – $138K/yr
Posted today

NTT DATA is hiring a Senior Data Engineering Support Engineer to anchor enterprise big data operations and maintain the availability, performance, and cost efficiency of production data pipelines. The role handles complex incidents, performance and cluster optimization, automation, data governance, documentation, and technical mentorship.

Posted
Today
Source
careers.nttdata.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Critical Incident & Escalation Management: Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs
  • Deep-Dive Performance Tuning: Drive the resolution of complex distributed computing bottlenecks, including severe data skew, Out-Of-Memory (OOM) errors, improper partition pruning, and costly shuffle operations using Spark UI, logs, and Ganglia metrics
  • Cost & Cluster Optimization: Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend (e.g., configuring auto-scaling, spot instances, and choosing optimal driver/executor node types)
  • Governance & Reliability Frameworks: Oversee and troubleshoot data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog
  • Automation & SRE Mindset: Build robust, automated frameworks and scripts in Python/PySpark to automate data reconciliation, system health checks, and self-healing pipelines
  • Strategic Playbooks & Documentation: Author comprehensive, technical runbooks and post-mortem RCA documents. Establish SOPs (Standard Operating Procedures) to reduce future incident resolution time (MTTR)
  • Mentorship & Leadership: Provide technical guidance to junior support engineers, conduct technical code reviews for hotfixes, and closely collaborate with core development teams to bridge the gap between engineering and operations

Qualifications

  • 7+ years of total experience in Data Engineering
  • 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production

Preferred

  • Advanced PySpark and Python engineering skills, with an explicit understanding of Spark architecture (lazy evaluation, DAG execution, execution memory vs. storage memory)
  • High proficiency in advanced SQL for complex analytical troubleshooting
  • Expert-level understanding of Delta Lake internals (ACID transactions, Time Travel, Liquid Clustering, Optimize/Z-Order commands)
  • Proven track record of diagnosing and resolving distributed data pipeline issues at scale, including thread dumps, memory leaks, and serialization errors
  • Strong architectural experience deploying and managing resources within major cloud networks (AWS, Azure, or GCP)
  • Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment via CI/CD pipelines
  • Active Databricks Certifications (e.g., Databricks Certified Data Engineer Professional)
  • Deep experience managing production environments utilizing Delta Live Tables (DLT) and advanced Unity Catalog rule engines
  • Delta Lake Time Travel
  • Unity Catalog Rule Engines

Required skills

AWSAzureCI/CD PipelinesData EngineeringDatabricksDatabricks CertificationDatabricks Unity CatalogDelta LakeDelta Live TablesGCPGit-Based WorkflowsPySparkPythonSQL

Full job description

NTT DATA is hiring a Senior Data Engineering Support Engineer to anchor enterprise big data operations and maintain the availability, performance, and cost efficiency of production data pipelines. The role handles complex incidents, performance and cluster optimization, automation, data governance, documentation, and technical mentorship.

Responsibilities

  • Critical Incident & Escalation Management: Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs
  • Deep-Dive Performance Tuning: Drive the resolution of complex distributed computing bottlenecks, including severe data skew, Out-Of-Memory (OOM) errors, improper partition pruning, and costly shuffle operations using Spark UI, logs, and Ganglia metrics
  • Cost & Cluster Optimization: Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend (e.g., configuring auto-scaling, spot instances, and choosing optimal driver/executor node types)
  • Governance & Reliability Frameworks: Oversee and troubleshoot data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog
  • Automation & SRE Mindset: Build robust, automated frameworks and scripts in Python/PySpark to automate data reconciliation, system health checks, and self-healing pipelines
  • Strategic Playbooks & Documentation: Author comprehensive, technical runbooks and post-mortem RCA documents. Establish SOPs (Standard Operating Procedures) to reduce future incident resolution time (MTTR)
  • Mentorship & Leadership: Provide technical guidance to junior support engineers, conduct technical code reviews for hotfixes, and closely collaborate with core development teams to bridge the gap between engineering and operations

Preferred

  • Advanced PySpark and Python engineering skills, with an explicit understanding of Spark architecture (lazy evaluation, DAG execution, execution memory vs. storage memory)
  • High proficiency in advanced SQL for complex analytical troubleshooting
  • Expert-level understanding of Delta Lake internals (ACID transactions, Time Travel, Liquid Clustering, Optimize/Z-Order commands)
  • Proven track record of diagnosing and resolving distributed data pipeline issues at scale, including thread dumps, memory leaks, and serialization errors
  • Strong architectural experience deploying and managing resources within major cloud networks (AWS, Azure, or GCP)
  • Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment via CI/CD pipelines
  • Active Databricks Certifications (e.g., Databricks Certified Data Engineer Professional)
  • Deep experience managing production environments utilizing Delta Live Tables (DLT) and advanced Unity Catalog rule engines
  • Python Engineering
  • Spark Architecture
  • Lazy Evaluation
  • DAG Execution
  • Spark Memory Management
  • Advanced SQL
  • Analytical Troubleshooting
  • Delta Lake Internals
  • ACID Transactions
  • Delta Lake Time Travel
  • Liquid Clustering
  • Delta Optimize
  • Z-Order
  • Distributed Data Pipelines
  • Pipeline Issue Diagnosis
  • Thread Dump Analysis
  • Memory Leak Diagnosis
  • Serialization Error Diagnosis
  • Cloud Resource Management
  • AWS
  • Azure
  • GCP
  • Git Workflows
  • Automated Testing
  • Secure Hotfix Deployment
  • CI/CD Pipelines
  • Databricks Certifications
  • Production Environment Management
  • Delta Live Tables
  • Unity Catalog Rule Engines
  • Problem-Solving

Benefits

  • May be eligible for incentive compensation based on individual and/or company performance.
  • Medical, dental, and vision insurance with an employer contribution.
  • Flexible spending or health savings account.
  • Life and AD&D insurance.
  • Short- and long-term disability coverage.
  • Paid time off.
  • Employee assistance.
  • Participation in a 401(k) program with company match.
  • Additional voluntary or legally required benefits.
  • Remote role

Company

  • IT and business services, consulting, modernization, and managed services
  • Tokyo, Tokyo, JPN
  • https://www.nttdata.com/jp/ja/
  • Founded in 1988
  • 10001+ employees

Qualifications

  • 7+ years of total experience in Data Engineering
  • 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production
  • Advanced PySpark and Python engineering skills, with an explicit understanding of Spark architecture (lazy evaluation, DAG execution, execution memory vs. storage memory)
  • High proficiency in advanced SQL for complex analytical troubleshooting
  • Expert-level understanding of Delta Lake internals (ACID transactions, Time Travel, Liquid Clustering, Optimize/Z-Order commands)
  • Proven track record of diagnosing and resolving distributed data pipeline issues at scale, including thread dumps, memory leaks, and serialization errors
  • Strong architectural experience deploying and managing resources within major cloud networks (AWS, Azure, or GCP)
  • Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment via CI/CD pipelines
  • Active Databricks Certifications (e.g., Databricks Certified Data Engineer Professional)
  • Deep experience managing production environments utilizing Delta Live Tables (DLT) and advanced Unity Catalog rule engines
  • 7+ years of total experience in Data Engineering.
  • 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production.

Required skills

  • AWS, Azure, CI/CD Pipelines, Data Engineering, Databricks, Databricks Certification, Databricks Unity Catalog, Delta Lake, Delta Live Tables, Distributed Data Pipeline Troubleshooting, GCP, Git-Based Workflows, PySpark, Python, SQL, Spark Architecture

Experience

  • 7+ years

Job details

Employment type

Full-time

Seniority level

7+ years

Date posted

Nov 20, 2026

Work mode

Remote

Job source

careers.nttdata.com

Company

NTT DATA

Information Technology & AI

IT and business services, consulting, modernization, and managed services

See all 6 jobs from NTT DATA →
  • United States
  • Information Technology & AI
  • 10001+ employees
  • Founded 1988
  • nttdata.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Data Engineer jobs

Dice

Today

New

Senior Data Engineer - Phoenix, AZ, St. Louis, MO, Nashville, TN, Kansas City, KS, Denver, CO.

TemporaryRemoteUnited StatesCompetitive

TechniPros is recruiting a Senior Data Engineer for an unnamed client. The role focuses on designing and building scalable cloud-native Lakehouse platforms and data pipelines to support enterprise analytics, AI/ML, and Generative AI initiatives.

Why similarSame roleDatabricksDelta Lake

Bright Vision Technologies

Today

New

Azure Data Engineer

Full-timeRemoteUnited States$115K – $165K/yr

Bright Vision Technologies is seeking an Azure Data Engineer to design, build, validate, and support data solutions for enterprise operations, analytics, and decision-making.

Why similarSame roleData EngineeringPython

Greystar

Today

New

Data Platform Engineer

Full-timeRemoteUnited States$125K – $150K/yr

Greystar Management Services, LLC is hiring a Data Platform Engineer to operate and govern its enterprise Databricks data platform on Microsoft Azure.

Why similarSame roleDelta LakePython

Get new Data Engineer jobs by email

Create job alert

Senior Data Engineering - Support Engineer

NTT DATA · $92K – $138K/yr

Apply on company site