← Back to jobsJob ID col-cmwf7u1jt0cn089hqvakzouzp

RelativityPosted 8 days ago

Product Manager - AI Evaluations Tooling

Full-time
Remote
4+ years
$115K – $173K/yr
Posted 8 days ago

Relativity is expanding its AI portfolio and building shared infrastructure to help teams evaluate and improve AI performance. The Product Manager will own the strategy and roadmap for the Evaluations Platform, including agentic testing, production monitoring, quality dashboards, SME workflows, and build-versus-buy decisions. The role partners with engineering, applied science, legal experts, and product teams to make AI quality measurable and actionable.

Posted
8 days ago
Source
kcura.wd1.myworkdayjobs.com
Sector
Product

Opens the employer's application page in a new tab

Responsibilities

  • Own the product vision, strategy, and roadmap for the Evaluations Platform across its pillars: offline evaluation pipeline, online evaluation and production monitoring, eval discovery and reuse, SME authoring and approval, and agentic evaluation
  • Serve every team building AI at Relativity. Run continuous discovery with applied scientists, application engineering teams, and product PMs to understand how each of them experiences AI quality today and design the platform so teams can measure and improve their own performance without routing every question through an applied scientist
  • Lead the orchestration of a world-class agentic testing system: trajectory evaluation, tool-call correctness, intermediate-state rubrics, long-running judge orchestration, and simulation of multi-step flows. Make it fast and safe to ship new agentic capabilities and make the platform a competitive advantage in how quickly Relativity can iterate on agents
  • Work closely with legal subject-matter experts to define the tasks our AI is evaluated against, what a correct outcome looks like, where the hard cases are, and how judgment criteria should be expressed so they can be encoded, scored, and reused across products. Build the SME authoring and approval workflow around how experts work
  • Own the team-facing quality dashboards. Every AI product team should be able to open a view and understand their own performance, where they stand against their quality bar, and what changed since last release, with no engineering help required
  • Lead build-vs-buy evaluations for tracing infrastructure and commercial eval tooling and run a recurring review as the vendor landscape shifts
  • Partner with engineering and applied science on judge calibration so LLM-as-judge is trustworthy enough to gate deployments
  • Clearly articulate technical tradeoffs to stakeholders ranging from applied scientists to attorneys to executive leadership

Qualifications

  • 4+ years of product management experience at a technology company, with at least 2 years on technical, platform, infrastructure, or data/ML products
  • Solid understanding of the software development lifecycle and agile practices
  • Experience conducting user discovery with technical audiences and translating findings into roadmap decisions
  • Excellent written and verbal communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences
  • Product Lifecycle Management (PLM)

Preferred

  • Experience with LLM-based products and a working understanding of how they are evaluated (rubrics, LLM-as-judge, offline vs. online evaluation)
  • Direct experience with agentic systems is a plus, not a requirement
  • Experience as a platform, developer-tools, or data/ML PM serving technical customers, or a demonstrated ability to learn a technical domain quickly and earn credibility with engineers and scientists
  • Comfort bringing non-engineering domain experts into technical workflows and translating their judgment into something a system can act on
  • Strong analytical instincts: sets metrics, tests assumptions, and changes course when the evidence says to
  • LLM Product Experience
  • Developer Tools Product Management
  • Technical Customer Experience

Required skills

Agile MethodologyLLM EvaluationLLM-as-JudgeProduct ManagementProduct StrategySoftware Development LifecycleStakeholder ManagementUser Discovery

Nice to have

Written and Verbal Communication

Full job description

Relativity is expanding its AI portfolio and building shared infrastructure to help teams evaluate and improve AI performance. The Product Manager will own the strategy and roadmap for the Evaluations Platform, including agentic testing, production monitoring, quality dashboards, SME workflows, and build-versus-buy decisions. The role partners with engineering, applied science, legal experts, and product teams to make AI quality measurable and actionable.

Responsibilities

  • Own the product vision, strategy, and roadmap for the Evaluations Platform across its pillars: offline evaluation pipeline, online evaluation and production monitoring, eval discovery and reuse, SME authoring and approval, and agentic evaluation
  • Serve every team building AI at Relativity. Run continuous discovery with applied scientists, application engineering teams, and product PMs to understand how each of them experiences AI quality today and design the platform so teams can measure and improve their own performance without routing every question through an applied scientist
  • Lead the orchestration of a world-class agentic testing system: trajectory evaluation, tool-call correctness, intermediate-state rubrics, long-running judge orchestration, and simulation of multi-step flows. Make it fast and safe to ship new agentic capabilities and make the platform a competitive advantage in how quickly Relativity can iterate on agents
  • Work closely with legal subject-matter experts to define the tasks our AI is evaluated against, what a correct outcome looks like, where the hard cases are, and how judgment criteria should be expressed so they can be encoded, scored, and reused across products. Build the SME authoring and approval workflow around how experts work
  • Own the team-facing quality dashboards. Every AI product team should be able to open a view and understand their own performance, where they stand against their quality bar, and what changed since last release, with no engineering help required
  • Lead build-vs-buy evaluations for tracing infrastructure and commercial eval tooling and run a recurring review as the vendor landscape shifts
  • Partner with engineering and applied science on judge calibration so LLM-as-judge is trustworthy enough to gate deployments
  • Clearly articulate technical tradeoffs to stakeholders ranging from applied scientists to attorneys to executive leadership

Preferred

  • Experience with LLM-based products and a working understanding of how they are evaluated (rubrics, LLM-as-judge, offline vs. online evaluation)
  • Direct experience with agentic systems is a plus, not a requirement
  • Experience as a platform, developer-tools, or data/ML PM serving technical customers, or a demonstrated ability to learn a technical domain quickly and earn credibility with engineers and scientists
  • Comfort bringing non-engineering domain experts into technical workflows and translating their judgment into something a system can act on
  • Strong analytical instincts: sets metrics, tests assumptions, and changes course when the evidence says to
  • LLM Product Experience
  • LLM Evaluation
  • Evaluation Rubrics
  • LLM-as-Judge
  • Offline Evaluation
  • Online Evaluation
  • Agentic Systems
  • Platform Product Management
  • Developer Tools Product Management
  • Data/ML Product Management
  • Technical Customer Experience
  • Technical Workflow Design
  • Expert Judgment Translation
  • Metrics Definition
  • Assumption Testing
  • Rapid Domain Learning
  • Credibility Building
  • Cross-Functional Collaboration
  • Technical Communication
  • Analytical Thinking
  • Evidence-Based Decision-Making
  • Adaptability

Benefits

  • Annual performance bonus
  • Long-term incentives
  • Remote/Hybrid

Company

  • Legal data intelligence software for litigation and investigations
  • Chicago, Illinois, USA
  • http://www.relativity.com/
  • Founded in 2001
  • 1001-5000 employees

Qualifications

  • 4+ years of product management experience at a technology company, with at least 2 years on technical, platform, infrastructure, or data/ML products
  • Solid understanding of the software development lifecycle and agile practices
  • Experience conducting user discovery with technical audiences and translating findings into roadmap decisions
  • Excellent written and verbal communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences
  • Agile Methodology
  • Cross-Functional Teamwork
  • Market Research
  • Market Strategy
  • Product Lifecycle
  • Product Lifecycle Management (PLM)
  • Product Management
  • Product Strategies
  • Stakeholder Management
  • Team Leadership
  • Experience with LLM-based products and a working understanding of how they are evaluated (rubrics, LLM-as-judge, offline vs. online evaluation)
  • Direct experience with agentic systems is a plus, not a requirement
  • Experience as a platform, developer-tools, or data/ML PM serving technical customers, or a demonstrated ability to learn a technical domain quickly and earn credibility with engineers and scientists
  • Comfort bringing non-engineering domain experts into technical workflows and translating their judgment into something a system can act on
  • Strong analytical instincts: sets metrics, tests assumptions, and changes course when the evidence says to
  • 4+ years of product management experience at a technology company, with at least 2 years on technical, platform, infrastructure, or data/ML products.

Required skills

  • Agile Methodology, LLM Evaluation, LLM-as-Judge, Product Management, Product Strategy, Software Development Lifecycle, Stakeholder Management, Technical Platform Product Management, User Discovery

Preferred skills

  • Written and Verbal Communication

Experience

  • 4+ years

Job details

Employment type

Full-time

Seniority level

4+ years

Sector · Sub-sector

Product · Product Management

Date posted

Nov 13, 2026

Work mode

Remote

Job source

kcura.wd1.myworkdayjobs.com

Company

Relativity

Product

Legal data intelligence software for litigation and investigations

See all 4 jobs from Relativity →
  • Chicago, Illinois, USA
  • Product
  • 1001-5000 employees
  • Founded 2001
  • relativity.com

Similar jobs

Matched on role, skills and location · live jobs only

See more AI Product Manager jobs

Meta

2d ago

New

Product Marketing Manager, Generative AI

Full-timeOn-siteMenlo Park, CA$190K – $258K/yr

Meta seeks a Product Marketing Manager to drive market requirements and go-to-market plans for its AI offerings, with a focus on Muse.

Why similarSame roleProduct Management

NVIDIA

2d ago

New

Senior Product Marketing Manager, Local AI

Full-timeOn-siteSanta Clara, CA$152K – $230K/yr

NVIDIA is hiring a technical, AI-first product marketing manager to define the narrative and lead go-to-market coordination for its on-device Local AI platform.

Why similarSame role

PointClickCare

2d ago

New

Outbound Product Manager - 12 month contract

TemporaryRemoteUnited States$122K – $152K/yr

PointClickCare is hiring an Outbound Product Manager to lead go-to-market execution and adoption for assigned products and initiatives.

Why similarSimilar titleProduct Management

Get new AI Product Manager jobs by email

Create job alert

Product Manager - AI Evaluations Tooling

Relativity · $115K – $173K/yr

Apply on company site