Chain of Thought: Introducing SEAL Showdown
Brad Kenstler, Head of Agent Capabilities and Environments, discusses SEAL Showdown with Bing Liu, Head of Research, and Janie Gu, Product Manager.
SEAL Showdown is a new public leaderboard – one built on real preferences, real people, and real-world use across practical categories like country, education level, age, language, and profession.
0:00 Introduction to Showdown
2:17 What makes Showdown unique
5:30 Showdown statistics and methodology
15:50 Preventing gaming and overfitting
18:20 Transparency and protecting user privacy
20:12 Dataset details
24:30 Surprising insights
28:44 Implications of Showdown to the research community
31:12 Future of the model evaluation landscape
36:38 Showdown's roadmap
Check out SEAL Showdown here: https://scale.com/showdown