Maximum Likelihood as Minimizing KL Divergence
While the Bayes' formula for the posterior probability or for parameters given the data is very general, there are some interesting special cases where that can be analyzed separately. Let's look at them in a sequence. The first special case arises when the model is a fixed one and for all. In this case, we can drop the conditioning on M in this formula. The Bayesian evidence, in this case, is simply some function of data only, and usually turns out to be irrelevant for the analysis of the most probable values of data. If in addition to fixing the model we additionally take flat prior, which is a prior that is a constant function of data, then maximization of the posterior probability is equivalent to maximization of probability to see the observed data given the model and a current value of parameter Theta. This method is known as the maximum likelihood estimation or MLE for short. The MLE is a very popular method of estimation of probabilistic models in both machine learning and statistics. As a method of model estimation, it offers a number of attractive properties.