Teach with ByteByteGoLogin

AI Evals in Practice: For Engineers & PMs

Live & hands-on

Manjeet SinghSenior Director @Salesforce

What you'll learn

Understand how AI evals work

  • Learn why unit tests and traditional QA break on non-deterministic systems
  • Measure AI quality like accuracy, safety, reliability, cost, and latency
  • Turn real failures into your first evals with error analysis
  • Take a multi-step agent trace and find where it went wrong in minutes, not days.

Build LLM judges you trust

  • Grade outputs automatically with LLM-as-a-Judge
  • Write pass/fail rubrics that behave like assertions
  • Validate every judge against human review before relying on it

Red-team agents before attackers do

  • Attack your own agent with prompt injection and jailbreaks
  • Automate red teaming with Promptfoo
  • Add guardrails that block attacks without blocking real users
  • Put your agent through a safety and adversarial review your security team takes seriously.

Build meaningful eval datasets

  • Build golden datasets from real traffic plus validated synthetic data
  • Version datasets like code so results stay reproducible
  • Score RAG pipelines with the RAGAS metrics

Evaluate agents on different dimensions

  • Test whether agents call the right tools with the right arguments
  • Judge the agent's full trajectory and final answer
  • Catch failures hiding in multi-agent handoffs

Run evals in production

  • Wire evals into CI/CD so regressions never merge
  • Monitor live traffic
  • Catch model and data drift

Ship a complete eval system

  • Take a capstone from untested agent to gated, monitored pipeline
  • Pick the cheapest model that still meets your quality bar
  • Turn every incident into a permanent regression test

Course Outline

Week 1
Wednesday, October 7, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

Eval Foundations and Error Analysis

Why AI Products Need Evals

  • Production failure case studies
  • Model benchmarks vs. product evals
  • Non-determinism and how to approach it
  • Traditional QA vs. AI evals
  • Evals as core infrastructure

The Fundamentals of AI Evaluation

  • The six quality dimensions: model quality, product behavior, safety, reliability, cost, latency
  • Model quality vs. product behavior evals
  • Cost metrics: per-task cost tracking, token and retry accounting
  • Latency metrics: time-to-first-token, span timing, percentile tracking
  • Eval harness architecture and pipeline components
  • Eval framework landscape (open-source and commercial)

Build a Real Agent

  • Create a real customer service agent so every run leaves a trace you can inspect.
  • Set up logging and observability that shows what the agent actually did.

Error Analysis

  • Traces vs. logs
  • The error analysis workflow
  • Failure taxonomies
  • Frequency-based prioritization
  • The eval flywheel: deploy → analyze → build dataset → improve → monitor

Assignment 1: Instrument and trace

  • Instrument the code assistant with the LangSmith SDK
  • Run 30 inputs: code-gen, doc lookup, multi-turn
  • Annotate traces end-to-end with pass/fail and notes

Assignment 2: Build a failure taxonomy

  • Cluster failures into 5+ binary trace codes with frequency counts
  • Map each code to a quality dimension
  • Report the top 3 failure modes and 1 unexplained anomaly
Eval Foundations and Error Analysis diagram
Week 2
Wednesday, October 14, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

Designing Reliable LLM Judges

Evaluation Methods

  • Evaluator types (code-based, traditional metrics, LLM-as-a-Judge, human eval)
  • Failure modes of traditional metrics
  • Evaluator selection: accuracy, cost, and latency trade-offs

Designing an LLM Judge

  • Binary pass/fail rubrics vs. Likert scales
  • Judge prompt anatomy
  • Single-criterion judges for root-cause isolation
  • The four judge types: tone/style, factual correctness, instruction following, task completion

Validation and Comparison

  • Judge–human agreement matrices
  • False-pass vs. false-fail analysis
  • Trust gates, false-pass rate, explainable failures
  • Multi-turn and conversation-level evaluation
  • Per-metric A/B evaluation

Assignment 1: Build and validate

  • Write 3 binary rubrics: code correctness, doc relevance, tool-call accuracy
  • Implement as LLM judges in DeepEval with strict output formats
  • Validate on the 40-label set and look into agreement and Cohen's κ

Assignment 2: Diagnose and compare

  • Run a diagnose-and-fix cycle
  • A/B test two prompt variants with per-metric deltas
  • Write a ship/no-ship decision
Designing Reliable LLM Judges diagram
Week 3
Wednesday, October 21, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

Safety and Adversarial Testing

The Adversarial Foundations

  • Adversarial evaluation vs. quality evaluation
  • The agent attack surface
  • Attack categories like prompt injection, jailbreaks, data leakage, harmful content, and bias

Attack Techniques

  • Direct vs. indirect prompt injection
  • Jailbreak techniques: role-play, hypotheticals, emotional framing, encoding
  • Data leakage targets
  • Bias testing: demographic prompt variation
  • Harmful-content testing

Building the Suite and Scoring Guardrails

  • Manual vs. automated red teaming
  • Automated red teaming with Promptfoo
  • Test suite schema: attack type, target, expected behavior
  • Guardrail confusion matrices
  • Guardrail composition

Assignment 1: Scan and triage

  • Run a Promptfoo baseline scan
  • Triage failing traces into a frequency × severity matrix
  • Craft 5 domain-specific attacks by hand

Assignment 2: Build and score guardrails

  • Build 3 guardrails (regex, LLM judge, or hybrid)
  • Score each on catch rate and over-block rate
  • Write a ship/no-ship verdict with numeric thresholds
  • Human-in-the-loop approval flows
Safety and Adversarial Testing diagram
Week 4
Wednesday, October 28, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

RAG Evaluation

Eval Data

  • The four dataset qualities: representativeness, diversity, label correctness, versioning
  • Golden-set schema
  • Criteria-based ground truth
  • Synthetic data generation and validation
  • Dataset versioning and pinning in LangSmith
  • Benchmark literacy (MMLU, HumanEval, SWE-bench)

RAG Evaluation

  • Component-wise evaluation: retrieval vs. generation
  • The four RAGAS metrics: faithfulness, answer relevance, context precision, context recall
  • Groundedness
  • RAGAS + LangSmith integration

Statistical Rigor

  • Sample size and margin of error
  • Paired testing
  • Bootstrap confidence intervals
  • Significance testing and p-values
  • Multiple comparisons and held-out re-testing
  • Repeatable harness design (pinned datasets, one config, reproducible runs, etc.)

Assignment 1: Build and version the dataset

  • Build a 150+ case golden set
  • Pin v1 in LangSmith for versioned A/B runs

Assignment 2: Measure and test

  • Wire the four RAGAS metrics into the harness
  • Find one groundedness failure
  • Run a paired bootstrap test
  • Ship the harness with one config, one command, and a reproducible report
RAG Evaluation diagram
Week 5
Wednesday, November 4, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

Multi-Agent Evaluation

Agent Eval Foundations

  • model, agent, and system
  • Agent failure modes: tool misuse, reasoning loops, memory drift, permission escalation, coordination conflict
  • Turn-level vs. conversation-level metrics

Tool Use and Trajectories

  • Tool-call scoring
  • Expected-action sets
  • Argument validation as structured-output evaluation
  • Trajectory evaluation
  • Property-based trajectory judging vs. exact-match diffs

Multi-Agent Coordination

  • Handoff evaluation (completeness, fidelity, routing)
  • Cascading failures in agent pipelines
  • Model→system score gaps as interaction-failure signals

Assignment 1: Score tools and trajectories

  • Build 15+ expected-action records with order and extra-call policies
  • Score tool use per dimension
  • Build a trajectory evaluator
  • Flag one fragile pass

Assignment 2: Find interaction failures

  • Wire the code-gen review handoff
  • Produce a three-layer report for model, agent, and system
  • Document 3 interaction failures with handoff root causes
Multi-Agent Evaluation diagram
Week 6
Wednesday, November 11, 20265-7 PM PT Live Session7-8 PM PT Live Q&A

Production Eval Infrastructure

Eval Gates in CI/CD

  • Eval checkpoints (PR check, nightly regression, pre-launch gate)
  • Eval gates in GitHub Actions
  • Gating policy: blocking vs. logged metrics
  • Fast PR subsets vs. full regression suites

Online Evaluation and Drift Detection

  • Online evaluation signals
  • Traffic sampling for eval cost control
  • Alerting policy
  • The drift taxonomy (data drift, concept drift, prompt/model drift)
  • Scheduled canary evaluation

Compounding Eval

  • The eval flywheel: from production failures to regression tests
  • Eval-driven model routing and cost optimization
  • Ownership and operating cadence

Assignment 1: Deploy the infrastructure

  • Integrate the Weeks 1–5 harness into GitHub Actions: PR check + nightly cron, gated on safety and correctness
  • Deploy production monitoring
  • Schedule a drift canary with alerting

Assignment 2: Operate and document

  • Document one flywheel cycle like incident → root cause → fix → regression test
  • Produce an eval-driven cost-routing recommendation
  • Write the ownership doc
Production Eval Infrastructure diagram
Optional Office Hours

Drop in with questions about the material or your project. Entirely optional.

Mon, Oct 12 · 6–7 PM PTMon, Oct 19 · 6–7 PM PTMon, Oct 26 · 6–7 PM PTMon, Nov 2 · 6–7 PM PTMon, Nov 9 · 6–7 PM PTMon, Nov 16 · 6–7 PM PT
Drop in with questions about the material or your project. Entirely optional. diagram

Meet Your Instructor

Manjeet Singh

Manjeet Singh

Manjeet Singh is a Senior Director of AI Product Management at Salesforce, where he leads products that make large-scale AI agents production-ready: agent evaluations, observability, and auto-optimization. He has over 25 years of experience as an engineer and AI product leader across startups and large enterprises.

In this course, Manjeet uses his skills as an AI builder and practitioner to present a practical playbook on how to build an evaluation framework that can be trusted, how to operationalize it at scale, and how to use it to ship differentiated AI products. You'll get the full loop, hands-on: build a real AI agent, find where it breaks, and improve it with evals, the same discipline that is needed to run agents in production at enterprise scale.

Salesforce

Prerequisites

  • A basic understanding of at least one programming language.
  • Familiarity with Git and GitHub
  • Python 3.10+ on any Mac, Windows, or Linux computer with at least 4 GB of RAM
  • An API key for Claude, OpenAI, Gemini, or another LLM provider. Expect approximately $10–$20 in API usage costs.

What You’ll Get

Live & Interactive Sessions

Learn directly from the instructor in real time. Ask questions, receive feedback, and stay engaged.

Peer Community

Stay motivated and accountable with a group of peers who are learning alongside you.

Certificate of Completion

Showcase your achievement on LinkedIn. Proof that you’ve leveled up with real-world skills.

The ByteByteGo Guarantee

If the membership isn’t the right fit, you can request a full refund within 7 days of purchase, as long as you haven’t completed a course.

FAQs

Every session is recorded, so you can catch up anytime that works for you.

The course typically requires around 4–7 hours per week, but you can adjust the pace to fit your schedule with recorded sessions.

Yes. Every session is recorded, so you can catch up anytime. Most of our students are working professionals, and the course is designed with that in mind.

Many members expense it through their company’s learning and development budget. We provide an invoice you can submit and an email template you can send to your manager. Get the email template →

Once you enroll, you’ll keep access to your cohort’s recordings and course materials even after the live sessions end, so you can revisit lessons anytime.

Reach out to live-courses@bytebytego.com. We’ll get back to you within 24 hours.