Jobs in Colorado 2026 (Now Hiring) - Smart Auto Apply
We've scanned millions of jobs. Simply select your favorites, and we can fill out the applications for you.
AI Evaluation Engineer
$134,800 - $200,000 / year
Posted 2 weeks ago

Adjunct Faculty: Economics
$1,297 - $3,753 / project
Posted 30+ days ago

Excavation Salesperson
Posted 30+ days ago

Customer Service Associate I
$15 - $16 / hour
Posted 30+ days ago

PRN Occupational Therapist
Posted 30+ days ago
Client Relationship Executive - Cyber Managed Services
$122,500 - $504,500 / year
Posted 2 weeks ago
Quality Assurance Technician (Peterson Space Force Base)
$24 - $27 / hour
Posted 6 days ago
Surgery Scheduler - Orthopedics, Highlands Ranch
$43,700 - $51,042 / year
Posted 30+ days ago
Assistant Coach - Girls Swimming - HS
Posted 30+ days ago
Diesel Technician - Pay Up To $58/Hr & Weekends Off!
$20 - $58 / hour
Posted 3 weeks ago
Owner & Medical Director Veterinarian - Littleton, CO
Posted 4 days ago
Sr Mechanical Engineer - Propulsion
$137,386 - $188,927 / year
Posted 1 week ago
Foia Analyst
Posted 30+ days ago
Cybersecurity IT Audit Manager
$123,000 - $160,000 / year
Posted 30+ days ago
Adobe Target Lead Engineer
undefined114,240 - undefined178,500 / year
Posted 30+ days ago
Financial Advisor - Greenwood Village And Surrounding Areas
Posted 30+ days ago
Senior Director Of News, Kmgh
$220,000 - $245,000 / year
Posted 30+ days ago
Sr. Realtime Software Engineer
$165,000 - $218,000 / year
Posted 6 days ago
Primary Care RN Case Manager (Open Rank: Entry - Senior)
$60,257 - $70,500 / year
Posted 2 weeks ago
Operations Associate, Westminster, #110
$15+ / hour
Posted 30+ days ago
AI Evaluation Engineer
$134,800 - $200,000 / year
Automate your job search with Sonara.
Submit 10x as many applications with less effort than one manual application.1
Reclaim your time by letting our AI handle the grunt work of job searching.
We continuously scan millions of openings to find your top matches.

Overview
Job Description
About Judi Health
Judi Health is an enterprise health technology company providing a comprehensive suite of solutions for employers and health plans, including:
- Judi Rx, a public benefit corporation delivering full-service pharmacy benefit management (PBM) solutions to self-insured employers,
- Judi Health, which offers full-service health benefit management solutions to employers, TPAs, and health plans, and
- Judi, the industry's leading proprietary Enterprise Health Platform (EHP), which consolidates all claim administration-related workflows in one scalable, secure platform.
Together with our clients, we're rebuilding trust in healthcare in the U.S. and deploying the infrastructure we need for the care we deserve. To learn more, visit www.judi.health.
Hybrid 3 days (offices in NYC, Denver, CO and Charlotte, NC area)
Position Summary
As an AI Evaluation Engineer at Judi Health, you will build the testing frameworks, metrics, and tooling used to assess the safety, reliability, and accuracy of AI models and autonomous agents in production. This role bridges the gap between model development and real‑world usage by translating ambiguous product goals into measurable quality targets.
We're looking for someone to lead evaluation end-to-end - from unit and integration testing to offline, online, and statistical evaluations of probabilistic systems. What we need is someone who can design and operate robust evaluation frameworks, partner with scientists and engineers, and ensure we can confidently answer questions like: "Did this change improve or degrade quality, safety, or user outcomes?"
What You'll Build
Evaluation & Quality Pipelines
- Build data evaluation pipelines that collect production conversations and agent interactions
- Reconstruct full sessions from traces, logs, recordings, and transcripts
- Apply labeling and scoring using human feedback signals (surveys, sentiment, outcomes) and automated evaluators (e.g., LLM‑as‑judge)
Continuous Quality & Safety Benchmarking
- Own weekly and on‑demand automated evaluation runs against staging and production
- Define benchmarks that track accuracy, reliability, and safety‑related signals
- Produce trend dashboards that clearly answer: "Did this deploy change quality or risk?"
Unified Evaluation Framework
- Design and extend a standardized evaluation framework that supports multiple agent types and workflows
- Translate high‑level product expectations into concrete success criteria and metrics
- Ensure new agents and features can be evaluated consistently with minimal friction
Self Service Evaluation Tooling
- Build APIs and internal tools so data scientists and engineers can go from "interesting scenario" to "included in the eval suite" quickly
- Enable scenario curation, dataset management, and eval execution without deep infrastructure knowledge
Experiment Tracking & Visibility
- Provide shared visibility into prompt, model, and agent experiments
- Enable reproducibility and comparison across runs so teams can build on each other's work instead of operating in silos
Position Responsibilities:
Data Engineering
- Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback)
- Implement complex data stitching and session reconstruction logic
- Manage dataset versioning, provenance, and lifecycle
Platform & Observability
- Develop dashboards and monitoring tools for AI quality metrics
- Integrate evaluations into CI/CD pipelines for scheduled and gated runs
- Implement alerting on quality and safety signals, not just infrastructure health
AI / ML Evaluation Tooling
- Apply and extend LLM‑as‑judge evaluation patterns
- Design metrics and scoring approaches suitable for stochastic, non‑deterministic systems
- Use tools like LangSmith to track runs, traces, experiments, and evaluation results
Collaboration
- Partner closely with data science, engineering, and product teams
- Translate between research goals, product intent, and engineering constraints
- Help define what "good" looks like for AI behavior in production
- Advocate for strong developer experience and usability in the tools you build
Required Qualifications
- 4+ years of experience in data engineering, ML engineering, or software engineering
- Bachelor's or Master's degree in Computer Science, Machine Learning, or a related quantitative field
- Strong proficiency in Python
- Experience building and maintaining production data pipelines
- Strong SQL skills
- Experience working with at least one cloud platform (AWS preferred)
Nice‑to‑Haves
- Prior work on LLM or agent evaluation infrastructure
- Familiarity with designing metrics for safety, reliability, or quality in AI systems
- Experience with voice or call‑center data (audio, transcripts, sentiment)
- Experience with browser automation tools (e.g., Playwright) for end‑to‑end evals
- Deep SQL expertise
New York, NY Salary Range
$161,600—$200,000 USD
Denver, CO Salary Range
$148,400—$185,000 USD
Charlotte, NC Salary Range
$134,800—$168,500 USD
All employees are responsible for adherence to the Judi Health Code of Conduct including the reporting of non-compliance. This position description is designed to be flexible, allowing management the opportunity to assign or reassign duties and responsibilities as needed to best meet organizational goals.
We provide equal employment opportunities to all employees and applicants for employment and prohibit discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, medical condition, genetic information, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
By submitting an application, you agree to the retention of your personal data for consideration for a future position at Judi Health. More details about Judi Health's privacy practices can be found at https://www.judi.health/legal/privacy-policy.
Automate your job search with Sonara.
Submit 10x as many applications with less effort than one manual application.
