← Back to jobsJob ID col-cmwp5jsdc060cd8has77sje4w

FICOPosted today

Software Engineer - LLM Applications (Python, Java)New

Full-time
Remote
$105K – $165K/yr
Posted today

FICO is hiring a Software Engineer to own the quality, safety, and performance of its AI-powered consumer features. The role combines Python and Java backend engineering with production LLM experience, focusing on prompts, evaluations, guardrails, model upgrades, and tools for analysts and QA engineers.

Posted
Today
Source
fico.wd1.myworkdayjobs.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Own the system prompts, tool descriptions, and evaluation prompts behind Personalized Credit Insights and the AI chat assistant, improving them using production evidence
  • Build and grow the evaluation framework: turn production failures into test cases, run regression suites before every prompt or model change, and report results the business can trust
  • Measure and improve LLM-based evaluator reliability by comparing against human-labeled data and tracking precision, recall, and agreement over time
  • Design, tune, and monitor guardrails, including input and output classifiers, PII redaction, prompt-injection defenses, and financial-advice compliance rules defined with Legal and Compliance
  • Lead model upgrades by re-running evaluations, comparing quality, cost, and latency, and shipping safely through A/B tests
  • Reduce cost and latency with prompt caching, model routing, and context management
  • Build tools for Conversation Analysts and QA engineers, such as transcript viewers, labeling workflows, and dashboards
  • Partner with Analytics, Legal, and Compliance on fairness and bias evaluation of AI outputs
  • Leverage AI coding assistants and agentic workflows (Claude Code, Cursor, GitHub Copilot) to accelerate delivery, with the judgment to review and refine their output

Qualifications

  • Bachelor's or Master's in Computer Science, Engineering, or a related field, with demonstrated professional software engineering experience, strong Python, and the ability to read and contribute to Java (Spring Boot) microservices
  • Proven experience building and operating LLM-powered applications in production, including prompt design, tool calling, structured outputs, and streaming
  • Demonstrated experience building evaluations for LLM systems: test sets, regression suites, and LLM-as-judge approaches, including measuring evaluator quality against human labels
  • Working knowledge of Amazon Bedrock or a comparable managed LLM platform, and AWS services such as S3, Lambda, and DynamoDB
  • Proficient SQL and comfort analyzing production data in a data warehouse such as Amazon Redshift
  • Production ownership, including code review, CI/CD, release management, and on-call support, within Agile/Scrum teams
  • Strong written and verbal communication, with the ability to explain evaluation results and tradeoffs to product, legal, and executive stakeholders

Preferred

  • Experience with guardrail tooling, LLM security testing, RAG and retrieval evaluation, fairness and bias evaluation, model migrations, observability tools such as Datadog, BI tools such as Tableau, or regulated domains (FCRA, ECOA, UDAAP, CFPB guidance) is a plus
  • Retrieval-Augmented Generation (RAG)

Required skills

AWS LambdaAmazon BedrockAmazon DynamoDBAmazon RedshiftAmazon S3JavaLLM Application DevelopmentLLM EvaluationPrompt DesignPythonSQLSpring Boot

Nice to have

Written and Verbal Communication

Full job description

FICO is hiring a Software Engineer to own the quality, safety, and performance of its AI-powered consumer features. The role combines Python and Java backend engineering with production LLM experience, focusing on prompts, evaluations, guardrails, model upgrades, and tools for analysts and QA engineers.

Responsibilities

  • Own the system prompts, tool descriptions, and evaluation prompts behind Personalized Credit Insights and the AI chat assistant, improving them using production evidence
  • Build and grow the evaluation framework: turn production failures into test cases, run regression suites before every prompt or model change, and report results the business can trust
  • Measure and improve LLM-based evaluator reliability by comparing against human-labeled data and tracking precision, recall, and agreement over time
  • Design, tune, and monitor guardrails, including input and output classifiers, PII redaction, prompt-injection defenses, and financial-advice compliance rules defined with Legal and Compliance
  • Lead model upgrades by re-running evaluations, comparing quality, cost, and latency, and shipping safely through A/B tests
  • Reduce cost and latency with prompt caching, model routing, and context management
  • Build tools for Conversation Analysts and QA engineers, such as transcript viewers, labeling workflows, and dashboards
  • Partner with Analytics, Legal, and Compliance on fairness and bias evaluation of AI outputs
  • Leverage AI coding assistants and agentic workflows (Claude Code, Cursor, GitHub Copilot) to accelerate delivery, with the judgment to review and refine their output

Preferred

  • Experience with guardrail tooling, LLM security testing, RAG and retrieval evaluation, fairness and bias evaluation, model migrations, observability tools such as Datadog, BI tools such as Tableau, or regulated domains (FCRA, ECOA, UDAAP, CFPB guidance) is a plus
  • Guardrail Tooling
  • LLM Security Testing
  • Retrieval-Augmented Generation (RAG)
  • Retrieval Evaluation
  • Fairness Evaluation
  • Bias Evaluation
  • Model Migration
  • Observability Tools
  • Datadog
  • Business Intelligence Tools
  • Tableau
  • FCRA
  • ECOA
  • UDAAP
  • CFPB Guidance

Benefits

  • An inclusive culture strongly reflecting our core values: Act Like an Owner, Delight Our Customers and Earn the Respect of Others.
  • The opportunity to develop professionally through valuable learning experiences.
  • Benefits and rewards programs that encourage you to bring your best every day and be recognized for doing so.
  • A people-first work environment offering work/life balance, employee resource groups, and social events to promote interaction and camaraderie.
  • Work from home.

Company

  • Analytics software and decision intelligence for businesses
  • Bozeman, Montana, USA
  • http://www.fico.com/
  • Founded in 1956
  • 1001-5000 employees

Qualifications

  • Bachelor's or Master's in Computer Science, Engineering, or a related field, with demonstrated professional software engineering experience, strong Python, and the ability to read and contribute to Java (Spring Boot) microservices
  • Proven experience building and operating LLM-powered applications in production, including prompt design, tool calling, structured outputs, and streaming
  • Demonstrated experience building evaluations for LLM systems: test sets, regression suites, and LLM-as-judge approaches, including measuring evaluator quality against human labels
  • Working knowledge of Amazon Bedrock or a comparable managed LLM platform, and AWS services such as S3, Lambda, and DynamoDB
  • Proficient SQL and comfort analyzing production data in a data warehouse such as Amazon Redshift
  • Production ownership, including code review, CI/CD, release management, and on-call support, within Agile/Scrum teams
  • Strong written and verbal communication, with the ability to explain evaluation results and tradeoffs to product, legal, and executive stakeholders
  • Experience with guardrail tooling, LLM security testing, RAG and retrieval evaluation, fairness and bias evaluation, model migrations, observability tools such as Datadog, BI tools such as Tableau, or regulated domains (FCRA, ECOA, UDAAP, CFPB guidance) is a plus

Required skills

  • AWS Lambda, Amazon Bedrock, Amazon DynamoDB, Amazon Redshift, Amazon S3, Java, LLM Application Development, LLM Evaluation, Prompt Design, Python, SQL, Spring Boot

Preferred skills

  • Written and Verbal Communication

Job details

Employment type

Full-time

Seniority level

Not stated

Date posted

Nov 20, 2026

Work mode

Remote

Job source

fico.wd1.myworkdayjobs.com

Company

FICO

Information Technology & AI

Analytics software and decision intelligence for businesses

See all 2 jobs from FICO →
  • Bozeman, Montana, USA
  • Information Technology & AI
  • 1001-5000 employees
  • Founded 1956
  • fico.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Python Engineer jobs

FICO

Today

New

Software Engineering – Data Engineer (Python & Cloud Databases)

Full-timeRemoteUnited States$84K – $132K/yr

FICO is hiring a Data Engineer for the myFICO engineering team. The role owns cloud databases, builds and operates Python data pipelines, prepares data for ML and AI applications, and supports analytics and data-driven decision-making.

Why similarSame roleAmazon RedshiftPython

Availity

Today

New

Software Engineer II

Full-timeRemoteUnited StatesCompetitive

Availity develops revenue cycle and related business solutions for healthcare organizations. The Software Engineer II develops and maintains payer customer implementations for the Authorizations suite, focusing primarily on backend development with some…

Why similarSimilar titleJavaBackend Engineering

M3 USA

Today

New

Software Engineer

Full-timeRemoteUnited StatesCompetitive

PracticeMatch, an M3 company, is hiring a Software Engineer to build and maintain features for its physician recruiting products.

Why similarSimilar titlePythonBackend Engineering

Get new Python Engineer jobs by email

Create job alert

Software Engineer - LLM Applications (Python, Java)

FICO · $105K – $165K/yr

Apply on company site