# Simple explanation of maximum likelihood estimation

I was watching a video from code emporium, here he describes Maximum Likelihood Estimation (MLE) as to finding weights which maximizes the probability of seeing the training data (D = [X,Y])

$$\theta^{MLE} =argmax \{ P(Y|X;\theta) \}$$

Let's consider our training data is XOR data

Input   Output
A   B
0   0   0
0   1   1
1   0   1
1   1   0


and say this data's real probability distribuiton is the blue one $$P(Y|X)$$ and our current model's (randomly initialized weights) probability disbribution is the red one $$P(Y|X;\theta_{Random})$$ Then the aim of Maximum Likelihood Estimation is to push the weights from randomly initialitzed weights to the real weights which approximately "describes" our real data distribution.

MLE = $$\{ P(Y|X;\theta_{Random})\rightarrow P(Y|X;\theta_{Real})\approx P(Y|X)\}$$

Question 1: Is my thinking right; Is this the correct approach to describe MLE?

In gradient descent, to train a model we have to minimize a cost function but in MLE we are maximizing it. To solve this in here he talks about Negative Log Likelihood which is a negative added to the log converted MLE?

Question 2: How is adding a negative to log converted MLE maximises MLE?

My intiution tells me instead of learning the real distribution $$P(Y|X;\theta_{Real})$$ this should have pushed away the weights to some random distribution?

## 1 Answer

Maximum likelihood is a method for estimating parameters by maximizing the probability of the observed data. The main ingredients are:

• The data: $$D=(X,Y)$$
• The model parameters: $$\theta$$
• The model that relates data to the parameters: $$P(D|\theta)$$ (which can be written differently, depending on the situation: $$P(Y|X, \theta)$$ is one of the possibilities.)

One would typically name the probability of observing data given parameters likelihood and write it as $$L(\theta|D)=P(D|\theta)$$. This is just a change of notation. One then maximizes the likelihood in respect to the values of the parameters: $$\begin{equation} \hat{\theta}=\mathrm{argmax}_\theta L(\theta|D). \end{equation}$$

Quite often the likelihood is a product of many identical functions for different data points. In this case it is mathematically more convenient to maximize its logarithm, i.e. the log-likelihood: $$LL(\theta|D) = \log L(\theta|D)$$. The likelihood and its logarithm has the same maximum, since logarithm is a monotonous function. Finally, maximizing a function is the same as minimizing its negative, therefore minimizing a negative log-likelihood is the same as maximizing the log-likelihood, which is the same as maximizing the likelihood. This is important since the optimization algorithms are often spelled explicitly only for minimization (or only for maximization).