← Back to jobsJob ID col-cmwpd98ug0aved8haj24lohob

NVIDIAPosted 1 day ago

Software Engineer, ML PerformanceNew

Full-time
On-site
12+ years
$224K – $356K/yr
Posted 1 day ago

NVIDIA is hiring a Software Engineer, ML Performance to optimize performance across distributed AI training and inference workloads. The role analyzes bottlenecks and regressions across the software and hardware stack, defines fleet-wide GPU performance metrics, evaluates automated diagnoses, and works with partners to drive improvements.

Posted
1 day ago
Source
nvidia.wd5.myworkdayjobs.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Analyze the performance of distributed training and inference workloads throughout the entire stack: model and framework, data loading and checkpointing, compiler and kernels, CUDA and NCCL, host and network. Identify the bottleneck, resolve it or suggest a solution, and provide the supporting evidence
  • Define the performance metrics tracked across a fleet of GPUs. Break them down into factors engineers need to debug. Decide which signals the platform collects first
  • Drive the analysis of regressions: segment workloads across machines, versions, and configurations to isolate the cause, and turn the method into analysis others can reuse
  • Evaluate automated diagnoses: judge whether a finding is accurate, identify misleading results and missed problems, and build the labeled cases that keep the analysis honest
  • Work closely with build partners on their real workloads, carry findings through to improvements, and feed what the product is missing back into the roadmap

Qualifications

  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience
  • 12+ years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production
  • Demonstrated performance work across more than one layer of the stack, with a clear account of the baseline, evidence, alternative explanations and validated outcome
  • Experience diagnosing performance across many jobs, machines or tenants, not only tuning a single workload
  • Enough familiarity with framework, runtime and compiler interfaces to say where a signal should be measured, even when someone else writes the instrumentation
  • Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and the effects of the measurement itself, and know when the evidence is insufficient
  • Strong programming and debugging skills, demonstrated by maintainable analysis code, tests or repeatable investigation procedures, including when using automated development tools
  • Clear communication of findings, trade-offs and remaining uncertainty to the people who own the workload
  • Proficiency with Python and CUDA

Preferred

  • Improved operation of both training and inference
  • Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang or Dynamo
  • Experience moving between application, system software, driver or hardware roles, and the full-stack view that comes with it
  • Statistical judgment applied to noisy, multi-node fleet-scale performance data. Ideally, experience with counter-factual analysis and segmentation of workloads for diagnosing regressions
  • Built debugging or performance-analysis tools used by other engineers, with evidence that they made investigations more effective
  • Performance analysis tool development

Required skills

CUDADistributed Performance AnalysisJAXNCCLPyTorchTensorRT-LLMVLLM

Full job description

NVIDIA is hiring a Software Engineer, ML Performance to optimize performance across distributed AI training and inference workloads. The role analyzes bottlenecks and regressions across the software and hardware stack, defines fleet-wide GPU performance metrics, evaluates automated diagnoses, and works with partners to drive improvements.

Responsibilities

  • Analyze the performance of distributed training and inference workloads throughout the entire stack: model and framework, data loading and checkpointing, compiler and kernels, CUDA and NCCL, host and network. Identify the bottleneck, resolve it or suggest a solution, and provide the supporting evidence
  • Define the performance metrics tracked across a fleet of GPUs. Break them down into factors engineers need to debug. Decide which signals the platform collects first
  • Drive the analysis of regressions: segment workloads across machines, versions, and configurations to isolate the cause, and turn the method into analysis others can reuse
  • Evaluate automated diagnoses: judge whether a finding is accurate, identify misleading results and missed problems, and build the labeled cases that keep the analysis honest
  • Work closely with build partners on their real workloads, carry findings through to improvements, and feed what the product is missing back into the roadmap

Preferred

  • Improved operation of both training and inference
  • Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang or Dynamo
  • Experience moving between application, system software, driver or hardware roles, and the full-stack view that comes with it
  • Statistical judgment applied to noisy, multi-node fleet-scale performance data. Ideally, experience with counter-factual analysis and segmentation of workloads for diagnosing regressions
  • Built debugging or performance-analysis tools used by other engineers, with evidence that they made investigations more effective
  • inference optimization
  • PyTorch
  • JAX
  • Megatron
  • NCCL
  • TensorRT-LLM
  • vLLM
  • SGLang
  • Dynamo
  • Full-stack systems expertise
  • Application-layer expertise
  • Systems software expertise
  • Driver software expertise
  • Hardware expertise
  • Statistical analysis
  • Fleet-scale performance analysis
  • Counterfactual analysis
  • Workload segmentation
  • Performance regression diagnosis
  • Debugging tool development
  • Performance analysis tool development

Benefits

  • Eligible for equity

Company

  • Accelerated computing platforms for AI and compute-intensive workloads
  • Santa Clara, California, USA
  • https://www.nvidia.com/
  • Founded in 1993
  • 10001+ employees

Qualifications

  • Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience
  • 12+ years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production
  • Demonstrated performance work across more than one layer of the stack, with a clear account of the baseline, evidence, alternative explanations and validated outcome
  • Experience diagnosing performance across many jobs, machines or tenants, not only tuning a single workload
  • Enough familiarity with framework, runtime and compiler interfaces to say where a signal should be measured, even when someone else writes the instrumentation
  • Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and the effects of the measurement itself, and know when the evidence is insufficient
  • Strong programming and debugging skills, demonstrated by maintainable analysis code, tests or repeatable investigation procedures, including when using automated development tools
  • Clear communication of findings, trade-offs and remaining uncertainty to the people who own the workload
  • Proficiency with Python and CUDA
  • Improved operation of both training and inference
  • Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang or Dynamo
  • Experience moving between application, system software, driver or hardware roles, and the full-stack view that comes with it
  • Statistical judgment applied to noisy, multi-node fleet-scale performance data. Ideally, experience with counter-factual analysis and segmentation of workloads for diagnosing regressions
  • Built debugging or performance-analysis tools used by other engineers, with evidence that they made investigations more effective
  • 12+ years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production.

Required skills

  • CUDA, Distributed Performance Analysis, GPU Training and Inference Performance Optimization, JAX, NCCL, PyTorch, TensorRT-LLM, VLLM

Preferred skills

  • Improved operation of both training and inference

Experience

  • 12+ years

Job details

Employment type

Full-time

Seniority level

12+ years

Date posted

Nov 20, 2026

Work mode

On-site

Job source

nvidia.wd5.myworkdayjobs.com

Company

NVIDIA

Information Technology & AI

Accelerated computing platforms for AI and compute-intensive workloads

See all 24 jobs from NVIDIA →
  • Santa Clara, California, USA
  • Information Technology & AI
  • 10001+ employees
  • Founded 1993
  • nvidia.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Machine Learning Engineer jobs in Santa Clara

Apple

1d ago

New

ML Data Operations Engineer

Full-timeOn-siteCupertino, CA$150K – $225K/yr

Apple is hiring an ML Data Operations Engineer to support internal data collection efforts for consumer machine learning features.

Why similarSame roleCupertino

Abridge

1d ago

New

Member of Technical Staff, Machine Learning Platform

Full-timeHybridSan Francisco, CA$221K – $260K/yr

Abridge is hiring for its machine learning platform team to build and optimize the inference infrastructure powering its machine learning models.

Why similarSame rolePyTorchSan Francisco

Uber

1d ago

New

Sr Machine Learning Engineer

Full-timeHybridSan Francisco, CA$202K – $224K/yr

Uber is a global technology company operating mobility, delivery, and membership services through products such as Uber and Uber Eats.

Why similarSame rolePyTorchSan Francisco

Get new Machine Learning Engineer jobs in Santa Clara by email

Create job alert

Software Engineer, ML Performance

NVIDIA · $224K – $356K/yr

Apply on company site