← Back to jobsJob ID col-cmwf9hw3t0f0d89hqyuwkly6v

Ellipsis HealthPosted 8 days ago

Senior Backend Engineer, AI Evaluations

Full-time
Hybrid
5+ years
$175K – $250K/yr
Posted 8 days ago

Ellipsis Health is seeking a Senior Backend Engineer to build infrastructure and internal tooling for reliable, repeatable evaluation of AI systems in production. The role develops evaluation frameworks, debugging and observability tools, and platform capabilities in partnership with ML, engineering, and QA teams.

Posted
8 days ago
Source
jobs.ashbyhq.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools
  • Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails
  • Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling
  • Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use
  • Ensure evaluation tooling meets production standards for reliability, performance, and maintainability

Qualifications

  • 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling
  • Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines
  • Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar)
  • Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems
  • Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows
  • Strong debugging skills and a methodical approach to diagnosing production and evaluation issues
  • Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems
  • Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability

Preferred

  • Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)
  • Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling
  • Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery
  • Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks)
  • Background in digital health, care coordination, or patient-facing systems
  • Healthcare industry experience
  • HIPAA knowledge

Required skills

AI Evaluation PipelinesAPIsASRBackend SystemsCI/CD TestingConversational AIData PipelinesHIPAA and PHI HandlingJavaLLMsLangfusePythonTTSTypeScript

Full job description

Ellipsis Health is seeking a Senior Backend Engineer to build infrastructure and internal tooling for reliable, repeatable evaluation of AI systems in production. The role develops evaluation frameworks, debugging and observability tools, and platform capabilities in partnership with ML, engineering, and QA teams.

Responsibilities

  • Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools
  • Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails
  • Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling
  • Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use
  • Ensure evaluation tooling meets production standards for reliability, performance, and maintainability

Preferred

  • Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)
  • Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling
  • Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery
  • Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks)
  • Background in digital health, care coordination, or patient-facing systems
  • Complex ML systems
  • ASR
  • LLMs
  • TTS
  • Healthcare industry experience
  • Regulated environments
  • HIPAA knowledge
  • PHI handling
  • Conversational AI
  • Voice agents
  • Multi-turn dialogue
  • Latency management
  • Error recovery
  • LLM observability
  • LLM evaluation tools
  • Langfuse
  • Prompt evaluation
  • Digital health
  • Care coordination
  • Patient-facing systems

Benefits

  • 401(k) matching
  • Health, vision, and dental insurance
  • Very flexible paid time off

Company

  • Emotionally intelligent AI care management for healthcare
  • San Francisco, California, USA
  • http://www.ellipsishealth.com/
  • Founded in 2017
  • 51-200 employees

Qualifications

  • 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling
  • Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines
  • Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar)
  • Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems
  • Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows
  • Strong debugging skills and a methodical approach to diagnosing production and evaluation issues
  • Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems
  • Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability
  • Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)
  • Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling
  • Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery
  • Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks)
  • Background in digital health, care coordination, or patient-facing systems
  • 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling.

Required skills

  • AI Evaluation Pipelines, APIs, ASR, Backend Systems, CI/CD Testing, Conversational AI, Data Pipelines, HIPAA and PHI Handling, Java, LLMs, Langfuse, Python, TTS, TypeScript

Experience

  • 5+ years

Job details

Employment type

Full-time

Seniority level

5+ years

Date posted

Nov 13, 2026

Work mode

Hybrid

Job source

jobs.ashbyhq.com

Company

Ellipsis Health

Information Technology & AI

Emotionally intelligent AI care management for healthcare

  • San Francisco, California, USA
  • Information Technology & AI
  • 51-200 employees
  • Founded 2017
  • ellipsishealth.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Backend Engineer jobs in San Francisco

#Backend Engineering Opportunities

1d ago

New

#Backend Engineering Opportunities | Backend Engineer & Lead Backend Engineer, AI Platform

Full-timeRemoteUnited States$140K – $240K/yr

Lakeview Loan Servicing is hiring Backend Engineers and Lead Backend Engineers for its AI Platform Backend Engineering team.

Why similarSame roleJavaRemote

Scribe

1d ago

New

Senior Backend Engineer

Full-timeRemoteUnited States$175K – $240K/yr

Scribe is hiring a Senior Backend Engineer to own a product area end to end, from design through post-launch results. The role builds and improves backend features, performance, and reliability while collaborating across product and engineering teams.

Why similarSame rolePythonRemote

SentiLink

1d ago

New

Senior Software Engineer, Platform/Backend

Full-timeRemoteUnited States$170K – $230K/yr

SentiLink builds identity verification and fraud-risk products for financial institutions. The Senior Software Engineer will build and enhance the API platform powering these products, collaborating across product, engineering, and security to deliver…

Why similarSame rolePythonRemote

Get new Backend Engineer jobs in San Francisco by email

Create job alert

Senior Backend Engineer, AI Evaluations

Ellipsis Health · $175K – $250K/yr

Apply on company site