Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction
Correction: At 30:42 I write "X = Y". They're not equal, what I meant to say is "X and Y are identically distributed".
The variance is a measure of how spread out a distribution is. In order to estimate the variance, one takes a sample of n points from the distribution, and calculate the average square deviation from the mean.
However, this doesn't give a good estimate of the variance of the distribution. The best estimate, however, is obtained when dividing by n-1 instead of n.
WHY!?!?!?!?!?!?!?
In this video, we dig deeper into why the variance calculation should be divided by n-1 instead of by n. For this, we use an alternate definition of the variance, which doesn't use the mean in its calculation.
[0:00] Introduction and Bessel's Correction
Introducing Bessel's Correction and why we divide by \( n-1 \) instead of \( n \) to estimate variance.
[0:12] Introduction to Variance Calculation
Explaining the premise of calculating variance and introducing the concept of estimating variance using a sample instead of the entire population.
[1:01] Definition of Variance
Defining variance as a measure of how much values deviate from the mean and outlining the basic steps of variance calculation.
[1:52] Introduction to Bessel's Correction
Discussing why we divide by \( n-1 \) when calculating variance and introducing Bessel's Correction.
[2:35] Challenges of Bessel's Correction
Sharing personal challenges in understanding the rationale behind Bessel's Correction and discussing my research process on the topic.
[3:20] Alternative Definition of Variance
Presenting an alternative definition of variance to aid in understanding Bessel's Correction and expressing curiosity about its presence in the literature.
[4:45] Quick Recap of Mean and Variance
Briefly revisiting the concepts of mean and variance, demonstrating how they are calculated with examples, and explaining how variance reflects different distributions.
[7:05] Sample Mean and Variance Estimation
Explaining the challenges of estimating the mean and variance of a distribution using a sample and discussing why sample variance is not a good estimate.
[8:49] Bessel's Correction and Why \( n-1 \) is Used
Explaining how Bessel's Correction provides a better estimate of variance and why we divide by \( n-1 \) instead of \( n \). Emphasizing the importance of making a correct variance estimate.
[10:51] Why Better Estimation Matters?
Discussing why the original estimate is poor and why making a better estimate is crucial. Explaining the significance of sample mean as a good estimate.
[13:02] Issues with Variance Estimation
Illustrating the problems with variance estimation and demonstrating with examples why using the correct mean is essential for accurate estimates. Explaining the accuracy of estimates made using \( n-1 \).
[15:04] Introduction to Correcting the Estimate
Discussing the underestimated variance and the need for correction in estimation.
[15:57] Adjusting the Variance Formula
Explaining the adjustment in the variance formula by changing the denominator from \( n \) to \( n - 1 \).
[16:22] Calculation Illustration
Demonstrating the calculation process of variance with the adjusted formula using examples.
[16:57] Better Estimate with Bessel's Correction
Discussing how the corrected estimate provides a more accurate variance estimation.
[18:24] New Method for Variance Calculation
Introducing a new method for calculating variance without explicitly calculating the mean.
[20:06] Understanding the Relation between Variance and Variance
Explaining the relationship between variance and variance, and how they are related mathematically.
[21:52] Demonstrating a Bad Calculation
Illustrating a flawed method for calculating variance and explaining the need for correction.
[23:37] The Role of Bessel's Correction
Explaining why removing unnecessary zeros in variance calculation leads to better estimates, equivalent to Bessel's Correction.
[25:08] Summary of Estimation Methods
Summarizing the difference between the flawed and corrected estimation methods for variance.
[26:02] Importance of Bessel's Correction
Emphasizing the significance of Bessel's Correction for accurate variance estimation, especially with smaller sample sizes.
[30:19] Mathematical Proof of Variance Relationship
Providing two proofs of the relationship between variance and variance, highlighting their equivalence.
[35:24] Acknowledgments and Conclusion
Thanks @mkan543 for the summary!