← Back to jobsJob ID col-cmwpedlls0bp7d8ha0o3kqojt

DoubleVerifyPosted 1 day ago

Senior AI EngineerNew

Full-time
Hybrid
5+ years
Competitive
Posted 1 day ago

DoubleVerify is seeking a Senior AI Engineer to build AgentOS, an agentic AI platform for conversational experiences across its Pinnacle advertising analytics suite. The role focuses on developing AI agents, integrating LLMs and analytics capabilities, and improving agent quality and platform integrations.

Posted
1 day ago
Source
doubleverify.wd5.myworkdayjobs.com
Sector
Information Technology & AI

Opens the employer's application page in a new tab

Responsibilities

  • Design, build, and maintain AI agents within the AgentOS framework using architectures based on MCP (Model Context Protocol), LangChain, LangGraph, and related tooling
  • Develop new MCP servers and tools that expose Pinnacle analytics capabilities — including metrics, dimensions, settings, and user data — to AI agents
  • Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration logic that handle complex, real-world advertising analytics queries
  • Build and maintain the chat UI integration layer, connecting AgentOS to the Pinnacle front-end experience
  • Work with major LLM providers (OpenAI, Anthropic, Google, and others) to select, prompt, and integrate models appropriate for specific agent tasks
  • Develop and own prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques
  • Implement Retrieval-Augmented Generation (RAG) pipelines to ground AI responses in Pinnacle's proprietary data and documentation
  • Continuously optimize model performance through evals, latency tuning, and cost management
  • Build and maintain evaluation (evals) frameworks that measure agent accuracy, instruction-following, hallucination rates, and user satisfaction
  • Define quality benchmarks and run regression evals on every agent change before production deployment
  • Instrument agents with observability tooling (e.g., LangSmith, Weights & Biases, or custom tracing) to support debugging and continuous improvement
  • Integrate AgentOS with Pinnacle's existing platform services — including PIAM/Keycloak authentication, Settings APIs, and analytics data layers
  • Collaborate with backend teams to build and consume APIs that surface data cleanly and securely to AI agents
  • Participate in architecture reviews and contribute to technical design documents for new AI-powered features
  • Partner closely with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows
  • Work with QE teams to define testing strategies for AI features, including behavioral testing and adversarial prompting
  • Contribute to the team's shared AI engineering standards, best practices, and documentation

Qualifications

  • 5+ years of professional software engineering experience building production-grade systems
  • 2+ years of hands-on experience developing LLM-based applications or AI agent systems
  • Demonstrated experience shipping AI-powered features end-to-end, from prototype to production
  • Solid hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks
  • Familiarity with MCP or similar modular tool-integration protocols for AI agents (or strong appetite to ramp quickly)
  • Experience calling and integrating with LLM APIs (OpenAI, Anthropic Claude, Google Gemini, or similar), including streaming, tool-calling, and structured output patterns
  • Strong Python proficiency; ability to build clean, testable, production-quality backend services (FastAPI experience a plus)
  • Understanding of retrieval-augmented generation patterns, vector stores, and semantic search
  • Experience building or using eval frameworks to measure LLM/agent quality and regression-test AI behavior
  • Familiarity with cloud environments (GCP, AWS, or Azure) and containerization basics (Docker/Kubernetes)
  • Strong problem-solving skills with a bias toward shipping and iterating
  • Clear communicator — able to explain AI behavior and limitations to both technical peers and non-technical stakeholders
  • Comfortable operating with ambiguity in a fast-paced, evolving AI landscape
  • Collaborative team player who contributes actively to shared standards and code quality
  • BS/MS in Computer Science, Machine Learning, AI, or a related technical field

Preferred

  • Experience with MCP server development and tool-use patterns in production agentic systems
  • Familiarity with PIAM, Keycloak, or OAuth 2.0-based authentication patterns
  • Knowledge of advertising analytics, AdTech platforms, or BI tools (Looker, Snowflake, etc.)
  • Experience with modern front-end frameworks (React/Angular) for conversational UI features
  • Contributions to open-source AI projects or published work in AI/ML
  • Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust
  • Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies
  • OAuth 2.0 Authentication

Required skills

AWSAdvertising AnalyticsAngularAzureDockerFastAPIGCPKeycloakKubernetesLLM API IntegrationLLM Evaluation FrameworksLangChainLangGraphLooker

Full job description

DoubleVerify is seeking a Senior AI Engineer to build AgentOS, an agentic AI platform for conversational experiences across its Pinnacle advertising analytics suite. The role focuses on developing AI agents, integrating LLMs and analytics capabilities, and improving agent quality and platform integrations.

Responsibilities

  • Design, build, and maintain AI agents within the AgentOS framework using architectures based on MCP (Model Context Protocol), LangChain, LangGraph, and related tooling
  • Develop new MCP servers and tools that expose Pinnacle analytics capabilities — including metrics, dimensions, settings, and user data — to AI agents
  • Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration logic that handle complex, real-world advertising analytics queries
  • Build and maintain the chat UI integration layer, connecting AgentOS to the Pinnacle front-end experience
  • Work with major LLM providers (OpenAI, Anthropic, Google, and others) to select, prompt, and integrate models appropriate for specific agent tasks
  • Develop and own prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques
  • Implement Retrieval-Augmented Generation (RAG) pipelines to ground AI responses in Pinnacle's proprietary data and documentation
  • Continuously optimize model performance through evals, latency tuning, and cost management
  • Build and maintain evaluation (evals) frameworks that measure agent accuracy, instruction-following, hallucination rates, and user satisfaction
  • Define quality benchmarks and run regression evals on every agent change before production deployment
  • Instrument agents with observability tooling (e.g., LangSmith, Weights & Biases, or custom tracing) to support debugging and continuous improvement
  • Integrate AgentOS with Pinnacle's existing platform services — including PIAM/Keycloak authentication, Settings APIs, and analytics data layers
  • Collaborate with backend teams to build and consume APIs that surface data cleanly and securely to AI agents
  • Participate in architecture reviews and contribute to technical design documents for new AI-powered features
  • Partner closely with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows
  • Work with QE teams to define testing strategies for AI features, including behavioral testing and adversarial prompting
  • Contribute to the team's shared AI engineering standards, best practices, and documentation

Preferred

  • Experience with MCP server development and tool-use patterns in production agentic systems
  • Familiarity with PIAM, Keycloak, or OAuth 2.0-based authentication patterns
  • Knowledge of advertising analytics, AdTech platforms, or BI tools (Looker, Snowflake, etc.)
  • Experience with modern front-end frameworks (React/Angular) for conversational UI features
  • Contributions to open-source AI projects or published work in AI/ML
  • Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust
  • Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies
  • MCP Server Development
  • Agentic Systems
  • Tool-Use Patterns
  • PIAM
  • Keycloak
  • OAuth 2.0 Authentication
  • Advertising Analytics
  • AdTech Platforms
  • BI Tools
  • Looker
  • Snowflake
  • React
  • Angular
  • Conversational UI Development
  • Open-Source AI Contributions
  • AI/ML Publications
  • Agent Evaluation Tools
  • DeepEval
  • LangSmith
  • Braintrust
  • Responsible AI
  • Bias Mitigation
  • Hallucination Reduction

Benefits

  • A bonus and equity
  • Comprehensive benefits package
  • Professional development and conference opportunities
  • Hybrid work flexibility from our New York City headquarters
  • A collaborative, innovative team culture with a strong emphasis on continuous learning and AI adoption

Company

  • Digital media measurement, analytics, and verification software platform
  • New York, New York, USA
  • http://www.doubleverify.com/
  • Founded in 2008
  • 501-1000 employees

Qualifications

  • 5+ years of professional software engineering experience building production-grade systems
  • 2+ years of hands-on experience developing LLM-based applications or AI agent systems
  • Demonstrated experience shipping AI-powered features end-to-end, from prototype to production
  • Solid hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks
  • Familiarity with MCP or similar modular tool-integration protocols for AI agents (or strong appetite to ramp quickly)
  • Experience calling and integrating with LLM APIs (OpenAI, Anthropic Claude, Google Gemini, or similar), including streaming, tool-calling, and structured output patterns
  • Strong Python proficiency; ability to build clean, testable, production-quality backend services (FastAPI experience a plus)
  • Understanding of retrieval-augmented generation patterns, vector stores, and semantic search
  • Experience building or using eval frameworks to measure LLM/agent quality and regression-test AI behavior
  • Familiarity with cloud environments (GCP, AWS, or Azure) and containerization basics (Docker/Kubernetes)
  • Strong problem-solving skills with a bias toward shipping and iterating
  • Clear communicator — able to explain AI behavior and limitations to both technical peers and non-technical stakeholders
  • Comfortable operating with ambiguity in a fast-paced, evolving AI landscape
  • Collaborative team player who contributes actively to shared standards and code quality
  • BS/MS in Computer Science, Machine Learning, AI, or a related technical field
  • Experience with MCP server development and tool-use patterns in production agentic systems
  • Familiarity with PIAM, Keycloak, or OAuth 2.0-based authentication patterns
  • Knowledge of advertising analytics, AdTech platforms, or BI tools (Looker, Snowflake, etc.)
  • Experience with modern front-end frameworks (React/Angular) for conversational UI features
  • Contributions to open-source AI projects or published work in AI/ML
  • Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust
  • Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies

Required skills

  • AWS, Advertising Analytics, Angular, Azure, Docker, FastAPI, GCP, Keycloak, Kubernetes, LLM API Integration, LLM Evaluation Frameworks, LangChain, LangGraph, Looker, Model Context Protocol (MCP), OAuth 2.0, Python, React, Retrieval-Augmented Generation (RAG), Snowflake, Vector Stores

Experience

  • 5+ years

Job details

Employment type

Full-time

Seniority level

5+ years

Date posted

Nov 20, 2026

Work mode

Hybrid

Job source

doubleverify.wd5.myworkdayjobs.com

Company

DoubleVerify

Information Technology & AI

Digital media measurement, analytics, and verification software platform

See all 2 jobs from DoubleVerify →
  • New York, New York, USA
  • Information Technology & AI
  • 501-1000 employees
  • Founded 2008
  • doubleverify.com

Similar jobs

Matched on role, skills and location · live jobs only

See more AI Engineer jobs in New York

WTW

1d ago

New

AI Solutions Engineer

Full-timeHybridNew York, NYCompetitive

WTW is hiring an AI Solutions Engineer for its Willis Neuron Technology team, supporting the Risk & Analytics consulting practices of Willis, a WTW business.

Why similarSame roleDockerNew York

Yotta Systems Inc

1d ago

New

Generative AI Engineer

Full-timeRemoteUnited StatesCompetitive

Yotta Systems Inc is seeking a Generative AI Engineer to develop AI applications using Python, LLMs, and retrieval-augmented generation. The role involves building AI agents and integrating REST APIs.

Why similarSame roleAWSRemote

just now

1d ago

New

Gen AI Engineer

Full-timeHybridNew York, NYCompetitive

Suntek Corporations Inc is seeking a Gen AI Engineer to design, develop, and productionize enterprise AI applications and workflows.

Why similarSame roleLangGraphNew York

Get new AI Engineer jobs in New York by email

Create job alert

Senior AI Engineer

DoubleVerify · Competitive

Apply on company site