← Back to jobsJob ID col-cmvv3nnf602y942b9ul8runo7

StreetIDPosted 30+ days ago

Artificial Intelligence Engineer

Full-time
Remote
Competitive
Posted 30+ days ago

StreetID is seeking a Senior AI Evaluations Engineer to build and operate evaluation frameworks that assess the accuracy, safety, reliability, and production readiness of agentic AI systems. The role works closely with AWS Professional Services and enterprise customers, primarily in financial services, to develop evaluation capabilities for multi-agent workflows using Amazon Bedrock.

Posted
30+ Days Ago
Source
linkedin.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • AI Evaluation Frameworks: Architect, develop, and maintain end-to-end evaluation frameworks for agentic AI systems, including offline regression testing, online monitoring, live-traffic evaluation, and A/B testing
  • Rubrics & Scoring: Design quality rubrics, scoring architectures, and evaluation standards across complex multi-agent workflows
  • LLM-as-Judge / Agent-as-Judge: Develop automated evaluation pipelines assessing task completion, tool-call accuracy, agent trajectories, groundedness, and safety
  • Python Engineering: Build scalable, production-grade Python applications and evaluation services supporting AI model testing, benchmarking, and performance monitoring
  • Evaluation Infrastructure: Own evaluation infrastructure from architecture through production deployment, integrating CI/CD pipelines, automated regression testing, observability, and monitoring
  • Agentic AI Development: Work with frameworks such as LangGraph, Strands, and CrewAI to evaluate complex agent interactions and multi-step workflows
  • AWS Integration: Leverage Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, and IAM to support secure, scalable evaluation systems
  • Technical Leadership: Collaborate with AWS architects, engineering teams, and enterprise customers, providing guidance on AI evaluation methodologies and production readiness

Qualifications

  • Expert-level Python programming with demonstrated experience building production-grade AI applications, backend services, or evaluation frameworks
  • Expert-level (Level 400) experience with LLMs, generative AI systems, and agentic AI architectures
  • Hands-on experience designing and implementing evaluation frameworks for LLM-powered agents, including offline/on-demand regression testing and online/continuous live-traffic scoring
  • Strong understanding of automated scoring, evaluation rubrics, and assessment at the session, response, and tool-call levels
  • Experience measuring task completion, tool-call accuracy, trajectory quality, groundedness, and safety
  • Hands-on experience with LangGraph, Strands, CrewAI, or similar agent orchestration frameworks
  • Practical experience with Amazon Bedrock, Bedrock Guardrails, CloudWatch, Lambda, S3, and IAM
  • Proven ability to architect, deploy, and support scalable, reliable AI applications in enterprise environments
  • Must hold an active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate certification

Preferred

  • Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing
  • Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems
  • Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development
  • Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics
  • Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring
  • Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives
  • Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements
  • Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications
  • Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients
  • A/B Testing
  • Agent-as-a-Judge Architectures
  • Human-in-the-Loop Evaluation
  • Reusable AI Evaluation Frameworks
  • Large Application Portfolio Delivery

Required skills

AI Benchmarking GAIAAI Benchmarking SWE-benchAI Benchmarking tau-benchAgentic AIEvaluation Metrics and ScoringFinancial Services AILLM Evaluation FrameworksLLM-as-Judge EvaluationLangGraph, Strands, and CrewAIPythonRAG Evaluation

Full job description

StreetID is seeking a Senior AI Evaluations Engineer to build and operate evaluation frameworks that assess the accuracy, safety, reliability, and production readiness of agentic AI systems. The role works closely with AWS Professional Services and enterprise customers, primarily in financial services, to develop evaluation capabilities for multi-agent workflows using Amazon Bedrock.

Responsibilities

  • AI Evaluation Frameworks: Architect, develop, and maintain end-to-end evaluation frameworks for agentic AI systems, including offline regression testing, online monitoring, live-traffic evaluation, and A/B testing
  • Rubrics & Scoring: Design quality rubrics, scoring architectures, and evaluation standards across complex multi-agent workflows
  • LLM-as-Judge / Agent-as-Judge: Develop automated evaluation pipelines assessing task completion, tool-call accuracy, agent trajectories, groundedness, and safety
  • Python Engineering: Build scalable, production-grade Python applications and evaluation services supporting AI model testing, benchmarking, and performance monitoring
  • Evaluation Infrastructure: Own evaluation infrastructure from architecture through production deployment, integrating CI/CD pipelines, automated regression testing, observability, and monitoring
  • Agentic AI Development: Work with frameworks such as LangGraph, Strands, and CrewAI to evaluate complex agent interactions and multi-step workflows
  • AWS Integration: Leverage Amazon Bedrock, Guardrails, CloudWatch, Lambda, S3, and IAM to support secure, scalable evaluation systems
  • Technical Leadership: Collaborate with AWS architects, engineering teams, and enterprise customers, providing guidance on AI evaluation methodologies and production readiness

Preferred

  • Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing
  • Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems
  • Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development
  • Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics
  • Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring
  • Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives
  • Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements
  • Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications
  • Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients
  • On-Demand SDK Evaluation
  • Online Monitoring
  • A/B Testing
  • Agent-as-a-Judge Architectures
  • Human-in-the-Loop Evaluation
  • Automated Scoring Systems
  • tau-bench
  • SWE-bench
  • GAIA
  • Benchmark Dataset Development
  • RAG Evaluation
  • Faithfulness Metrics
  • Context Precision Metrics
  • Recall Metrics
  • Evaluation Regression Testing
  • Distributed Tracing
  • Structured Logging
  • Production Monitoring
  • Java
  • Spring
  • Sybase
  • DB2
  • COBOL
  • AI-Driven Modernization
  • Financial Services Compliance
  • Trade Surveillance
  • Data Governance
  • Audit Requirements
  • Reusable AI Evaluation Frameworks
  • Large Application Portfolio Delivery
  • AWS Professional Services
  • AI Engineering
  • Enterprise Client Solutions

Company

  • Financial services job matchmaking platform
  • Brooklyn, New York, USA
  • http://www.streetid.com/
  • Founded in 2010
  • 11-50 employees

Qualifications

  • Expert-level Python programming with demonstrated experience building production-grade AI applications, backend services, or evaluation frameworks
  • Expert-level (Level 400) experience with LLMs, generative AI systems, and agentic AI architectures
  • Hands-on experience designing and implementing evaluation frameworks for LLM-powered agents, including offline/on-demand regression testing and online/continuous live-traffic scoring
  • Strong understanding of automated scoring, evaluation rubrics, and assessment at the session, response, and tool-call levels
  • Experience measuring task completion, tool-call accuracy, trajectory quality, groundedness, and safety
  • Hands-on experience with LangGraph, Strands, CrewAI, or similar agent orchestration frameworks
  • Practical experience with Amazon Bedrock, Bedrock Guardrails, CloudWatch, Lambda, S3, and IAM
  • Proven ability to architect, deploy, and support scalable, reliable AI applications in enterprise environments
  • Must hold an active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate certification
  • Experience with on-demand SDK evaluation runners, online monitoring, and A/B testing
  • Agent-as-a-judge architectures, human-in-the-loop (HITL) evaluation, and automated scoring systems
  • Familiarity with tau-bench, SWE-bench, GAIA, and custom benchmark dataset development
  • Experience evaluating retrieval-augmented generation systems using faithfulness, context precision, recall, and related quality metrics
  • Automated evaluation regression testing, distributed tracing, structured logging, and production monitoring
  • Familiarity with Java/Spring, Sybase, DB2, or COBOL in the context of AI-driven modernization initiatives
  • Experience supporting highly regulated financial services environments, including compliance, trade surveillance, data governance, and audit requirements
  • Experience delivering reusable AI evaluation frameworks across large application portfolios, including 50+ applications
  • Previous experience working with AWS Professional Services or delivering complex AI engineering solutions to enterprise clients

Required skills

  • AI Benchmarking GAIA, AI Benchmarking SWE-bench, AI Benchmarking tau-bench, AWS Generative AI or Machine Learning Certification, Agentic AI, Amazon Bedrock and AWS Services CloudWatch, Amazon Bedrock and AWS Services Guardrails, Amazon Bedrock and AWS Services IAM, Amazon Bedrock and AWS Services Lambda, Amazon Bedrock and AWS Services S3, Evaluation Metrics and Scoring, Financial Services AI, Generative AI and Large Language Models, LLM Evaluation Frameworks, LLM-as-Judge Evaluation, LangGraph, Strands, and CrewAI, Python, RAG Evaluation

Job details

Employment type

Full-time

Seniority level

Not stated

Date posted

Oct 17, 2026

Work mode

Remote

Job source

linkedin.com

Company

StreetID

Information Technology & AI

Financial services job matchmaking platform

  • Brooklyn, New York, USA
  • Information Technology & AI
  • 11-50 employees
  • Founded 2010
  • streetid.com

Similar jobs

Matched on role, skills and location · live jobs only

See more AI Engineer jobs

Veeam Software

Today

New

Finance AI Engineer

Full-timeRemoteUnited StatesCompetitive

Veeam Software is seeking a Finance AI Engineer to design, build, and deploy reliable AI solutions across Finance functions including Controllership, FP&A, Revenue, Tax, and Treasury.

Why similarSame rolePython

Khan Academy

1d ago

New

Senior Search & AI Discoverability Engineer (12 month contract, 40 hours/week, $85/hour)

Full-timeRemoteUnited StatesCompetitive

Khan Academy is hiring a Senior Search & AI Discoverability Engineer for its Marketing and Acquisition team.

Why similarSame role

SUPPLY

3d ago

AI & Marketing Technology Specialist

Full-timeRemoteUnited States$75K – $100K/yr

SUPPLY is an independent media agency hiring an AI & Marketing Technology Specialist to improve agency operations and develop technology-driven capabilities for its teams and clients.

Why similarSame rolePython

Get new AI Engineer jobs by email

Create job alert

Artificial Intelligence Engineer

StreetID · Competitive

Apply on company site