Maximum likelihood method
Enroll in the full Machine Learning in Python course at [email protected]
Sample likelihood is the probability of obtaining (among all possible samples) the exact input data that is available. In fact, if the parameters are independent, it is the product of the probability function or probability density function for each sample parameter to take its value in the sample.
For example, if we have a sample of 10 students, the sample likelihood function will describe the probabilities that the sample contains 0 girls, 1 girl, 2 girls, and so on.
Likelihood Maximization
An important practical use of sample likelihood is to maximize the likelihood of the actual value of the given sample: since such an event has occurred (its probability is 1), the likelihood function can be used to estimate the unknown parameters of the a priori known distribution law of the random variable.
For ease of working with partial derivatives, which allow one to find the maximum of a function, the logarithm of the likelihood function is used: the product is replaced by the sum.
Linear Regression
For a linear relationship between an unknown variable and known parameters, the use of the maximum likelihood method is reduced to the least squares method.
Using a Bayesian approach (maximizing the posterior probability) also allows for L2 regularization.
Binary Classification
An important consequence of the likelihood maximization method is that the expected value of events is calculated simply as the proportion of these events in the entire sample (the arithmetic mean).
For example, if we have 6 girls out of 10 students, then the estimated proportion of girls (for example, to predict the gender of the next students) will be exactly 0.6 (i.e., the next student is most likely to be a girl).
Exponential Distribution
This will also work well for predicting increasing intervals between events, assuming that the probability decreases exponentially over time. For example, a course enrolls 1 person per day, but sometimes 2 or 5 people show up. And on other days, no one enrolls.
Then, with good accuracy, the number of new students will be described by an exponential distribution with a mean of 1.
Nearest Neighbors Method
The maximum likelihood method is the basis for the nearest neighbors method in classification problems: we expect to obtain the class of an unknown object that is the same as that of most of its neighbors.
Naive Bayes
An improved version of this approach is also used in the naive Bayes model, which uses the posterior density maximization method—that is, it takes into account previous values of the unknown variable and, based on previous history and the actual sample values, produces a classification answer.
We'll discuss the practical applications of the maximum likelihood method in more detail in the relevant lessons.