Lead AI QA

Johnson Service GroupBoston, MA

Apply with Sonara

Automate your job search with Sonara.

Submit 10x as many applications with less effort than one manual application.¹

Reclaim your time by letting our AI handle the grunt work of job searching.

We continuously scan millions of openings to find your top matches.

Job Description

Our client is seeking a Senior-level AI Quality Assurance Leader specializing in Generative AI, LLM systems, and AI agents, responsible for defining and driving end-to-end quality strategy for scalable and responsible AI deployments. (Remote, USA) Must Have skills

Generative AI (GenAI) system testing 7+ years overall QA experience with hands-on GenAI validation
Large Language Models (LLMs) 5+ years experience validating LLM outputs for accuracy, safety, and bias
Retrieval-Augmented Generation (RAG) systems 5+ years experience testing pipeline performance and retrieval quality
Azure AI Foundry 3+ years experience testing Azure-based AI solutions
LangGraph 3+ years experience validating orchestration and multi-agent workflows
Test Automation Frameworks for AI Systems 5+ years experience building automated validation and evaluation pipelines
LangSmith evaluation workflows 3+ years experience in LLM evaluation and monitoring
Multi-agent AI architectures 3+ years experience testing agent coordination and decision logic
Responsible AI practices 5+ years experience in bias detection, safety validation, and compliance testing
Performance and load testing for AI systems 5+ years experience validating scalability of AI services
CI/CD integration for AI pipelines 5+ years experience embedding QA into deployment workflows
AI evaluation metrics design (BLEU, ROUGE, custom scoring, etc.) 3+ years experience defining quality benchmarks

Responsibilities Define and lead QA strategy for Generative AI pipelines, RAG systems, and multi-agent workflows Validate LLM outputs for accuracy, safety, bias, and performance across environments Oversee quality assurance of Azure AI Foundry-based AI solutions Ensure quality across LangGraph orchestration and LangSmith evaluation workflows Establish automated testing frameworks and AI-specific evaluation metrics Lead QA teams to ensure scalable, reliable, and responsible AI deployments#LI-SB1#D480

Automate your job search with Sonara.

Submit 10x as many applications with less effort than one manual application.

Apply with Sonara Apply manually

FAQs About Lead AI QA Jobs at Johnson Service Group

What is the work location for this position at Johnson Service Group?

This job at Johnson Service Group is located in Boston, MA, according to the details provided by the employer. Some roles may also include multiple work locations depending on the requirement.

What pay range can candidates expect for this role at Johnson Service Group?

Employer has not shared pay details for this role.

What employment applies to this position at Johnson Service Group?

The employer has not provided this information. This may be discussed during the hiring process.

What is the process to apply for this position at Johnson Service Group?

You can apply for this role at Johnson Service Group either through Sonara's automated application system, which helps you submit applications 10X faster with minimal effort, or by applying manually using the direct link on the job page.