Shaping AI Benchmarks with Together AI Co-Founder Percy Liang
In this episode of Gradient Dissent, Together AI co-founder and Stanford Associate Professor Percy Liang joins host, Lukas Biewald, to discuss advancements in AI benchmarking and the pivotal role that open-source plays in AI development.
🎙 Listen on Apple Podcasts : http://wandb.me/apple-podcasts
🎙 Listen on Spotify : http://wandb.me/spotify
He shares his development of HELM—a robust framework for evaluating language models. The discussion highlights how this framework improves transparency and effectiveness in AI benchmarks. Additionally, Percy shares insights on the pivotal role of open-source models in democratizing AI development and addresses the challenges of English language bias in global AI applications. This episode offers in-depth insights into how benchmarks are shaping the future of AI, highlighting both technological advancements and the push for more equitable and inclusive technologies.
✅ Subscribe to Weights & Biases → https://bit.ly/45BCkYz
⏳Timestamps:
00:00 Introduction
09:16 Discussion on benchmarking incentives with references to Kaggle.
10:31 Nuanced challenges of language model overfitting.
10:42 How to effectively choose AI models using leaderboards.
14:00 Advice on selecting models for company use.
16:45 Importance of having a robust model evaluation framework.
18:38 How individuals can contribute to AI benchmarking.
21:13 Discussion on specialized vs. generalized model performance.
27:36 Insights into the complexities of benchmarking AI agents.
29:18 Real-world applications and limitations of AI agents.
36:18 Percy reflects on his TED talk regarding open vs. closed source models.
42:02 Introduction to TogetherAI and its mission for open AI development.
47:11 Combining AI with music creation.
🎙 Get our podcasts on these platforms:
Apple Podcasts: http://wandb.me/apple-podcasts
Spotify: http://wandb.me/spotify
Google: http://wandb.me/gd_google
YouTube: http://wandb.me/youtube
Connect with Percy Liang:
/ percy-liang-717b8a
/ percyliang
Anticipatory Music Composer:
https://stanford.io/3y5VycN
HELM Blog Post Referenced:
https://crfm.stanford.edu/2024/02/18/...
Follow Weights & Biases:
/ weights_biases
/ wandb
Join the Weights & Biases Discord Server:
/ discord