← Back to jobsJob ID col-cmwpce07v0a8dd8ha21d0xd8e

UberPosted 1 day ago

Senior Program Manager, TechNew

Full-time
On-site
5+ years
$167K – $185K/yr
Posted 1 day ago

Uber is hiring a Senior Program Manager, Tech to lead LLM quality and evaluation programs across multilingual products and AI-powered experiences. The role defines model quality measures and evaluation frameworks, turns evaluation findings into improvements, and coordinates cross-functional teams on model fitness and launch readiness.

Posted
1 day ago
Source
jobs.uber.com
Sector
Public Sector and Government

Opens the employer's application page in a new tab

Responsibilities

  • Own the LLM quality and evaluation roadmap. Define program goals, milestones, dependencies, success metrics, and release criteria across models and use cases
  • Build scalable evaluation frameworks. Establish statistically defensible methods to assess accuracy such as recall and precision, relevance, completeness, instruction following, linguistic quality, and cultural appropriateness
  • Develop high-quality evaluation datasets. Lead the creation, annotation, adjudication, versioning, and maintenance of golden datasets that represent real user needs, challenging scenarios, and multilingual requirements
  • Evaluate model and system failures. Partner with technical teams to assess hallucinations, prompt-injection susceptibility, unsupported claims, context loss, and inconsistent outputs. Evaluate retrieval-augmented generation (RAG) for retrieval relevance, groundedness, and citation accuracy
  • Lead human and automated evaluation. Establish reviewer guidelines, calibrate LLM-as-a-judge systems against expert assessments, and measure agreement, false positives, false negatives, and severity classification
  • Drive model fitness decisions. Collaborating with LLMOpn in the Localization Technology team to compare models and configurations by locale, task, quality, cost, and latency to inform model selection, routing, and deployment
  • Close the quality improvement loop. Translate findings into prioritized improvements to prompts, retrieval, terminology, workflows, and model configurations. Partner with owners to verify fixes and prevent regressions
  • Operationalize continuous quality monitoring. Establish regression testing, production sampling, quality dashboards, escalation processes, and ongoing evaluation as models and user needs evolve
  • Lead cross-functional execution. Clarify ownership, resolve dependencies, communicate risks, and align stakeholders on quality tradeoffs and launch readiness

Qualifications

  • Minimum 5 years of professional experience in program management within a high-growth technology or large-scale enterprise environment
  • Bachelor's degree in Business Operations, Engineering, or a related field; or equivalent professional experience

Preferred

  • 8+ years of professional experience in program management within a high-growth technology or large-scale enterprise environment
  • Experience leading complex technical programs involving AI/ML, software platforms, quality engineering, or data-driven products
  • Demonstrated ability to deliver cross-functional programs with Engineering, Localization team, Data Scientist, Product, and Operations
  • Working understanding of LLM capabilities and limitations, including hallucinations, prompt engineering, prompt injection, RAG, context management, and tool use
  • Experience designing quality metrics, evaluation methodologies, human-review workflows, or experimental programs
  • Ability to interpret statistical results, assess dataset quality, and communicate the limitations of evaluation findings to leaderships and LLMOps
  • Strong analytical, stakeholder management, and written communication skills, with the ability to turn ambiguous requirements into executable plans
  • Experience evaluating multilingual LLMs, machine translation, conversational AI, or localization quality
  • Familiarity with MQM-based evaluation, terminology governance, and market-specific linguistic requirements
  • Experience with LLM evaluation platforms, automated regression pipelines, and production observability
  • Familiarity with SQL, Python, APIs, and data pipelines sufficient to investigate results and collaborate effectively with technical teams
  • Experience evaluating multimodal systems, including text, image, audio, or video
  • Experience establishing quality standards and launch criteria for AI systems operating at scale

Required skills

AI/ML Program ManagementLLM EvaluationMQM-Based EvaluationMachine TranslationMultilingual Localization QualityProgram ManagementPrompt EngineeringPrompt InjectionPythonRetrieval-Augmented Generation (RAG)SQLStakeholder ManagementStatistical Analysis

Full job description

Uber is hiring a Senior Program Manager, Tech to lead LLM quality and evaluation programs across multilingual products and AI-powered experiences. The role defines model quality measures and evaluation frameworks, turns evaluation findings into improvements, and coordinates cross-functional teams on model fitness and launch readiness.

Responsibilities

  • Own the LLM quality and evaluation roadmap. Define program goals, milestones, dependencies, success metrics, and release criteria across models and use cases
  • Build scalable evaluation frameworks. Establish statistically defensible methods to assess accuracy such as recall and precision, relevance, completeness, instruction following, linguistic quality, and cultural appropriateness
  • Develop high-quality evaluation datasets. Lead the creation, annotation, adjudication, versioning, and maintenance of golden datasets that represent real user needs, challenging scenarios, and multilingual requirements
  • Evaluate model and system failures. Partner with technical teams to assess hallucinations, prompt-injection susceptibility, unsupported claims, context loss, and inconsistent outputs. Evaluate retrieval-augmented generation (RAG) for retrieval relevance, groundedness, and citation accuracy
  • Lead human and automated evaluation. Establish reviewer guidelines, calibrate LLM-as-a-judge systems against expert assessments, and measure agreement, false positives, false negatives, and severity classification
  • Drive model fitness decisions. Collaborating with LLMOpn in the Localization Technology team to compare models and configurations by locale, task, quality, cost, and latency to inform model selection, routing, and deployment
  • Close the quality improvement loop. Translate findings into prioritized improvements to prompts, retrieval, terminology, workflows, and model configurations. Partner with owners to verify fixes and prevent regressions
  • Operationalize continuous quality monitoring. Establish regression testing, production sampling, quality dashboards, escalation processes, and ongoing evaluation as models and user needs evolve
  • Lead cross-functional execution. Clarify ownership, resolve dependencies, communicate risks, and align stakeholders on quality tradeoffs and launch readiness

Preferred

  • 8+ years of professional experience in program management within a high-growth technology or large-scale enterprise environment
  • Experience leading complex technical programs involving AI/ML, software platforms, quality engineering, or data-driven products
  • Demonstrated ability to deliver cross-functional programs with Engineering, Localization team, Data Scientist, Product, and Operations
  • Working understanding of LLM capabilities and limitations, including hallucinations, prompt engineering, prompt injection, RAG, context management, and tool use
  • Experience designing quality metrics, evaluation methodologies, human-review workflows, or experimental programs
  • Ability to interpret statistical results, assess dataset quality, and communicate the limitations of evaluation findings to leaderships and LLMOps
  • Strong analytical, stakeholder management, and written communication skills, with the ability to turn ambiguous requirements into executable plans
  • Experience evaluating multilingual LLMs, machine translation, conversational AI, or localization quality
  • Familiarity with MQM-based evaluation, terminology governance, and market-specific linguistic requirements
  • Experience with LLM evaluation platforms, automated regression pipelines, and production observability
  • Familiarity with SQL, Python, APIs, and data pipelines sufficient to investigate results and collaborate effectively with technical teams
  • Experience evaluating multimodal systems, including text, image, audio, or video
  • Experience establishing quality standards and launch criteria for AI systems operating at scale
  • 8+ years of professional experience in program management within a high-growth technology or large-scale enterprise environment.
  • Technical Program Leadership
  • AI/ML
  • Software Platforms
  • Quality Engineering
  • Data-Driven Products
  • Cross-Functional Collaboration
  • LLM Capabilities
  • LLM Limitations
  • Hallucination Analysis
  • Prompt Engineering
  • Prompt Injection
  • Retrieval-Augmented Generation
  • Context Management
  • LLM Tool Use
  • Quality Metrics
  • Evaluation Methodologies
  • Human-Review Workflows
  • Experimental Program Design
  • Statistical Interpretation
  • Dataset Quality Assessment
  • Evaluation Findings Communication
  • Multilingual LLM Evaluation
  • Machine Translation
  • Conversational AI
  • Localization Quality
  • MQM-Based Evaluation
  • Terminology Governance
  • Linguistic Requirements
  • LLM Evaluation Platforms
  • Automated Regression Pipelines
  • Production Observability
  • SQL
  • Python
  • APIs
  • Data Pipelines
  • Multimodal System Evaluation
  • Quality Standards
  • AI Launch Criteria
  • Analytical Skills
  • Stakeholder Management
  • Written Communication
  • Ambiguity Management

Benefits

  • Eligible to participate in Uber's bonus program.
  • May be offered an equity award.
  • All full-time employees are eligible to participate in a 401(k) plan.
  • Full remote work, if approved.

Company

  • Ride-sharing and delivery platform for flexible earnings
  • San Francisco, California, USA
  • https://www.uber.com/
  • Founded in 2009
  • 10001+ employees

Qualifications

  • Minimum 5 years of professional experience in program management within a high-growth technology or large-scale enterprise environment
  • Bachelor's degree in Business Operations, Engineering, or a related field; or equivalent professional experience
  • 8+ years of professional experience in program management within a high-growth technology or large-scale enterprise environment
  • Experience leading complex technical programs involving AI/ML, software platforms, quality engineering, or data-driven products
  • Demonstrated ability to deliver cross-functional programs with Engineering, Localization team, Data Scientist, Product, and Operations
  • Working understanding of LLM capabilities and limitations, including hallucinations, prompt engineering, prompt injection, RAG, context management, and tool use
  • Experience designing quality metrics, evaluation methodologies, human-review workflows, or experimental programs
  • Ability to interpret statistical results, assess dataset quality, and communicate the limitations of evaluation findings to leaderships and LLMOps
  • Strong analytical, stakeholder management, and written communication skills, with the ability to turn ambiguous requirements into executable plans
  • Experience evaluating multilingual LLMs, machine translation, conversational AI, or localization quality
  • Familiarity with MQM-based evaluation, terminology governance, and market-specific linguistic requirements
  • Experience with LLM evaluation platforms, automated regression pipelines, and production observability
  • Familiarity with SQL, Python, APIs, and data pipelines sufficient to investigate results and collaborate effectively with technical teams
  • Experience evaluating multimodal systems, including text, image, audio, or video
  • Experience establishing quality standards and launch criteria for AI systems operating at scale
  • Minimum 5 years of professional experience in program management within a high-growth technology or large-scale enterprise environment.

Required skills

  • AI/ML Program Management, LLM Evaluation, MQM-Based Evaluation, Machine Translation, Multilingual Localization Quality, Program Management, Prompt Engineering, Prompt Injection, Python, Retrieval-Augmented Generation (RAG), SQL, Stakeholder Management, Statistical Analysis

Experience

  • 5+ years

Job details

Employment type

Full-time

Seniority level

5+ years

Date posted

Nov 20, 2026

Work mode

On-site

Job source

jobs.uber.com

Company

Uber

Public Sector and Government

Ride-sharing and delivery platform for flexible earnings

See all 12 jobs from Uber →
  • San Francisco, California, USA
  • Public Sector and Government
  • 10001+ employees
  • Founded 2009
  • uber.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Program Manager jobs in San Francisco

EvenUp

1d ago

New

Technical Program Manager, Enterprise

Full-timeRemoteUnited States$145K – $166K/yr

EvenUp develops technology and AI products for personal injury law firms and victims. The Technical Program Manager will lead enterprise platform implementations, retiring legacy systems and deploying a next-generation solution for key clients.

Why similarSame roleProgram ManagementRemote

Public company

30+ days ago

Program Manager II, AI & Analytics, Rapid & Rural Logistics (R2L)

Full-timeOn-siteNew York, NY$82K – $143K/yr

Amazon is seeking a strategic Program Manager to lead AI enablement across its Rapid & Rural Logistics operational businesses.

Why similarSame roleProgram ManagementSQL

Sales enablement

2d ago

New

Enablement Program Manager (contract)

TemporaryRemoteUnited States$401K/yr

Airbnb is hiring an Enablement Program Manager to build and scale central enablement programs for its field teams.

Why similarSame roleProgram ManagementRemote

Get new Program Manager jobs in San Francisco by email

Create job alert

Senior Program Manager, Tech

Uber · $167K – $185K/yr

Apply on company site