About the Seminar
AI features don't fail like traditional software. They drift, hallucinate, change tone, and give different answers to the same question. Pass/fail doesn't work here. So how does QA do its job?
In this hands-on workshop you'll learn how to test AI-powered features - from chatbots to autonomous agents to AI-generated code. You'll define what "good" looks like for LLM responses, build golden datasets, and create quality scorecards you can automate. You'll take an agent apart into its deterministic and non-deterministic pieces and design tests for each. You'll test guardrails under adversarial conditions and explore the security risks and economics unique to AI testing.
Who Is This Seminar For?
This hands-on workshop is designed for tech professionals building, evaluating, or managing AI-powered applications. It is ideal for:
- QA & Automation Engineers / Tech Leads
- AI & Software Engineers building LLM features, autonomous agents, or RAG systems
- AI Product Managers
- QA Managers & Engineering Directors
Prerequisites
- Core Concepts: A foundational understanding of LLMs, prompts, and generative AI features (e.g., chatbots or agents).
- Technical Background: Basic experience with software testing methodologies, API testing, or test automation.
- Coding Experience: Familiarity with basic scripting (preferably Python) is highly recommended for the automation and evaluation sections.
Key Topics
Takeaways
- How to test AI features across chats, agents, and AI-generated code - what's different about each and what testing strategies apply where.
- How to define quality for LLM responses - what makes a response good, acceptable, or bad, what should never happen, and how to assess
- testability.
- How to build golden datasets, create quality scorecards, and automate them using techniques like LLM-as-a-judge for repeatable evaluation.
- How to test an agent end-to-end - separating deterministic from non-deterministic behavior and applying the right testing approach to each.
- How to test guardrails and ethical boundaries - verifying that the protections others built actually hold under adversarial conditions.
- How to account for AI-specific security risks and token economics in your testing strategy.