← Back to jobsJob ID col-cmwpeppop0bvwd8haoh7qakwe

BMC SoftwarePosted 1 day ago

Principal QA EngineerNew

Full-time
Remote
$153K – $255K/yr
Posted 1 day ago

BMC Software is hiring a Principal QA Engineer to build evaluation systems that determine whether its agentic AI is ready to ship. The role develops evaluation frameworks, datasets, scoring methods, and CI-based test harnesses, and uses evaluation results to improve AI quality, reliability, and responsible use.

Posted
1 day ago
Source
jobs.bmc.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications
  • Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently
  • Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices
  • Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship
  • Define quality metrics and release gates for customer-ready vs prototype
  • Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant
  • Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion)
  • Identify failure modes using observability/tracing; recommend reliability improvements
  • Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering
  • Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation

Qualifications

  • Bachelor's or master's in computer science, AI, or related field (or equivalent experience)
  • Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks (depth scales with level)
  • Understanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires
  • Python; building eval sets, rubrics, automated scoring; CI/CD integration
  • LLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar) as level requires
  • Rigorous, data-driven mindset; collaboration across DS, engineering, and product
  • Evidence of owning an eval suite that influenced a release decision
  • Builds consistent rubrics/datasets — not one-off manual spot checks only
  • Works independently on scoped quality problems
  • Past experience: Owned a defined evaluation suite or quality workflow for an AI feature and partnered with stakeholders on “good enough.”
  • Delivery evidence: Decision-ready quality report; evals integrated into a release or CI path for one product/team
  • Shared expectation: Independently owns well-scoped work from definition through delivery

Preferred

  • Ragas, DeepEval, promptfoo, or similar eval frameworks
  • AI safety/governance; enterprise/regulated environments
  • OpenShift/AWS; performance/load/reliability testing of AI apps
  • MLOps; open-weight models/local serving; MCP; Agile/Atlassian
  • AI application performance testing

Required skills

Agentic AICI/CDDeepEvalEmbedding RetrievalLLM Observability and TracingLLM-as-a-JudgeLangfuseOpenSearchOpenTelemetryPythonRAGRagasStatistical SamplingVector Databases

Full job description

BMC Software is hiring a Principal QA Engineer to build evaluation systems that determine whether its agentic AI is ready to ship. The role develops evaluation frameworks, datasets, scoring methods, and CI-based test harnesses, and uses evaluation results to improve AI quality, reliability, and responsible use.

Responsibilities

  • Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications
  • Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently
  • Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices
  • Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship
  • Define quality metrics and release gates for customer-ready vs prototype
  • Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant
  • Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion)
  • Identify failure modes using observability/tracing; recommend reliability improvements
  • Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering
  • Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation

Preferred

  • Ragas, DeepEval, promptfoo, or similar eval frameworks
  • AI safety/governance; enterprise/regulated environments
  • OpenShift/AWS; performance/load/reliability testing of AI apps
  • MLOps; open-weight models/local serving; MCP; Agile/Atlassian
  • Ragas
  • DeepEval
  • promptfoo
  • AI evaluation frameworks
  • AI safety
  • AI governance
  • Enterprise environments
  • Regulated environments
  • OpenShift
  • AWS
  • AI application performance testing
  • Load testing
  • Reliability testing
  • MLOps
  • Open-weight models
  • Local model serving
  • MCP
  • Agile
  • Atlassian

Benefits

  • This role can be based remotely in United States
  • Other rewards may include a variable plan and country specific benefits.

Company

  • Enterprise software for automation and IT management
  • Houston, Texas, USA
  • http://www.bmc.com/
  • Founded in 1980
  • 5001-10000 employees

Qualifications

  • Bachelor's or master's in computer science, AI, or related field (or equivalent experience)
  • Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks (depth scales with level)
  • Understanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires
  • Python; building eval sets, rubrics, automated scoring; CI/CD integration
  • LLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar) as level requires
  • Rigorous, data-driven mindset; collaboration across DS, engineering, and product
  • Evidence of owning an eval suite that influenced a release decision
  • Builds consistent rubrics/datasets — not one-off manual spot checks only
  • Works independently on scoped quality problems
  • Past experience: Owned a defined evaluation suite or quality workflow for an AI feature and partnered with stakeholders on “good enough.”
  • Delivery evidence: Decision-ready quality report; evals integrated into a release or CI path for one product/team
  • Shared expectation: Independently owns well-scoped work from definition through delivery
  • Ragas, DeepEval, promptfoo, or similar eval frameworks
  • AI safety/governance; enterprise/regulated environments
  • OpenShift/AWS; performance/load/reliability testing of AI apps
  • MLOps; open-weight models/local serving; MCP; Agile/Atlassian

Required skills

  • Agentic AI, CI/CD, DeepEval, Embedding Retrieval, Generative AI and LLM Evaluation, LLM Observability and Tracing, LLM-as-a-Judge, Langfuse, OpenSearch, OpenTelemetry, Python, RAG, Ragas, Statistical Sampling, Vector Databases, promptfoo

Preferred skills

  • AI safety/governance; enterprise/regulated environments, Ragas, DeepEval, promptfoo, or similar eval frameworks

Job details

Employment type

Full-time

Seniority level

Not stated

Date posted

Nov 20, 2026

Work mode

Remote

Job source

jobs.bmc.com

Company

BMC Software

Information Technology & AI

Enterprise software for automation and IT management

See all 2 jobs from BMC Software →
  • Houston, Texas, USA
  • Information Technology & AI
  • 5001-10000 employees
  • Founded 1980
  • bmc.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Machine Learning Engineer jobs

Cardinal Health

1d ago

New

AI/ML Engineer

Full-timeOn-siteUnited States$96K – $124K/yr

Cardinal Health is seeking a product-oriented AI/ML Engineer to design and deliver enterprise-scale Generative AI solutions.

Why similarSame rolePythonVector Databases

Bright Vision Technologies

1d ago

New

Machine Learning Engineer

Full-timeRemoteUnited States$115K – $165K/yr

Bright Vision Technologies is seeking a Machine Learning Engineer to develop, evaluate, and operationalize predictive and classification models for production applications.

Why similarSame rolePython

Social Media

1d ago

New

Machine Learning Engineer

Full-timeRemoteUnited States$186K – $260K/yr

Reddit is hiring a Machine Learning Engineer to build and operate production machine learning systems across its platform.

Why similarSame rolePython

Get new Machine Learning Engineer jobs by email

Create job alert

Principal QA Engineer

BMC Software · $153K – $255K/yr

Apply on company site