The Key Equation Behind Probability
Get 4 months extra on a 2 year plan here: https://nordvpn.com/artemkirsanov. It’s risk free with Nord’s 30 day money-back guarantee!
=====
My name is Artem, I'm a neuroscience PhD student at Harvard University.
🌎 Website and Social links: https://kirsanov.ai/
📥 "Receptive Field" neuro-newsletter: https://artemkirsanov.substack.com/
✨ Support me on Patreon to get access to Discord community: / artemkirsanov
=====
In this video, we explore the fundamental concepts that underlie probability theory and its applications in neuroscience and machine learning. We begin with the intuitive idea of surprise and its relation to probability, using real-world examples to illustrate these concepts.
From there, we move into more advanced topics:
1) Entropy – measuring the average surprise in a probability distribution.
2) Cross-entropy and the loss of information when approximating one distribution with another.
3) Kullback-Leibler (KL) divergence and its role in quantifying the difference between two probability distributions.
🕒 OUTLINE:
00:00 Introduction
02:00 Sponsor: NordVPN
04:07 What is probability (Bayesian vs Frequentist)
06:42 Probability Distributions
10:17 Entropy as average surprisal
13:53 Cross-Entropy and Internal models
19:20 Kullback–Leier (KL) divergence
20:46 Objective functions and Cross-Entropy minimization
24:22 Conclusion & Outro
=====
Special thanks to Crimson Ghoul for providing English subtitles!
Icons by https://www.freepik.com/
=====
Disclaimer: This channel is my personal project. The views and content expressed here are my own and are separate from my research role at Harvard University.
#probability #entropy #datascience
Description remastered: February 2026. Links & Bio updated; original context preserved.