Chain of Thought: Introducing Audio MultiChallenge
Brad Kenstler, Head of Agent Capabilities and Environments, discusses our first benchmark focused on audio with Advait Gosai, Research Engineer, Brian Jang, AI Product Manager, and Tyler Vuong, Research Scientist.
0:00 Overview of Audio MultiChallenge
3:48 Why are audio models different?
8:20 Audio vs Text MultiChallenge
12:06 Challenges on data and multi-turn
21:10 Benchmark design
25:27 Results
28:50 Rubric and Judge design
31:55 Where audio models struggle
38:30 S2S vs ASR TTS
45:20 Implications on Scaling
Check out the Leaderboard here: https://scale.com/leaderboard/audiomc