The 4 Things Every QA Engineer Must Test in AI Agents (Live Promptfoo Workshop)AI workshop 1
```
Traditional QA is breaking. If you input X you expect Y — but AI agents don't work that way. In this live, hands-on workshop I show senior QA engineers how to actually test AI agents and chatbots: the 4 things that matter (accuracy, safety, cost, latency), how to build real tests with Promptfoo + the Gemini API, how to catch jailbreaks and hallucinations, and why "90% correct" is the new pass criteria.
Open your laptop and build it right alongside me. By the end you'll have a working AI chatbot test suite and a testing strategy that won't blow up your token bill.
⚙️ WHAT YOU'LL LEARN
• Why the classic X → Y → Z model fails for AI products
• The 4 pillars of testing any AI agent: accuracy, safety, cost, latency
• How AI tokens & cost actually work (and why 1,000 random tests is a mistake)
• Building a Promptfoo config from scratch with a free Gemini API key
• LLM-rubric assertions: politeness, groundedness, hallucination checks
• Jailbreak / prompt-injection safety tests (the airline chatbot lawsuit)
• Running evals, reading the report, and debugging real failures live
• How to prioritise test cases by risk & token cost (reduce the blast radius)
• A first look at Mid-scene (vision-based UI testing) + a Playwright AI agent
🔗 TOOLS & LINKS
• Promptfoo: https://www.promptfoo.dev
• Google AI Studio (free Gemini API key): https://aistudio.google.com/app/apikey
• Mid-scene: https://midscenejs.com
• Config / repo: https://github.com/JagadeeshJayachand...
🧪 WHO THIS IS FOR
QA engineers, SDETs, and test automation folks moving from Selenium/Playwright into
testing AI agents, LLM apps, RAG systems and chatbots.
👉 The full Mid-scene + Playwright agent build is happening LIVE in the next session.
Hit SUBSCRIBE so you don't miss it, and register here: https://subscribe-forms.beehiiv.com/a...
⏱️ CHAPTERS
0:00 Intro — I ran a live AI workshop for QA engineers
0:35 The 3 AI tools we'll use (Promptfoo, Mid-scene, Playwright agent)
2:12 Why traditional QA testing breaks for AI (X → Y → Z)
5:00 The 4 things you must test: accuracy, safety, cost, latency
6:00 Safety & the airline chatbot refund lawsuit
7:53 How AI tokens & cost actually work
9:55 Latency — response speed
10:20 Recap of the 4 pillars + let's open Promptfoo
11:10 Install Promptfoo + prerequisites (Node)
17:22 Get a free Gemini API key
19:37 Set environment variables (Mac & Windows)
28:00 Build the Promptfoo config (YAML, customer-support agent)
34:11 Test scenarios: accuracy, jailbreaks & hallucinations (LLM rubric)
38:48 Run the eval (+ live debugging)
45:37 Read the report: what passed & what failed
51:12 Why AI tests are about probability (the 90% rule)
54:07 Test strategy: prioritise by risk & token cost
59:21 Promptfoo for red teaming, pen testing & RAG
1:01:07 Mid-scene: vision-based UI testing (live attempt)
1:07:49 Wrap-up & next session
#AItesting #QAengineer #Promptfoo #SoftwareTesting #AItools #LLM #TestAutomation #AIagents #Gemini #Chatbot