Software Engineer — AI Evaluation & Automation : 26

Akraya, Inc. is an IT staffing and talent solutions provider. The Software Engineer will build and scale systems for evaluating AI-powered software development tools, automate reproducible benchmarking processes, calibrate metrics against human judgment, and improve evaluation workflow reliability.

Responsibilities

• Develop and integrate evaluation systems for AI development tools
• Automate benchmark runs with versioned, reproducible environments
• Calibrate evaluation metrics against human judgment for accuracy
• Support performance benchmarking across quality and efficiency
• Analyze and improve the reliability of evaluation workflows

Benefits

• Remote work arrangement
• Possible contract extension
• Akraya's thriving work environment

Company

• Enterprise digital transformation, AI, product engineering, and data services
• San Francisco, California, USA
• http://akraya.com/
• Founded in 2001
• 201-500 employees

Qualifications

• Experience with AI coding tools and harnesses
• Proven evaluation and benchmarking experience
• Strong proficiency in Git and containerization with Docker
• Python, Java or JavaScript
• Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial

Required skills

• AI Coding Tools, AI Evaluation, API Integration, Docker, Git, Python, Software Benchmarking

Experience

• Mid level