1
$\begingroup$

I was watching a video from code emporium, here he describes Maximum Likelihood Estimation (MLE) as to finding weights which maximizes the probability of seeing the training data (D = [X,Y])

$\theta^{MLE} =argmax \{ P(Y|X;\theta) \}$

Let's consider our training data is XOR data

Input   Output
A   B
0   0   0
0   1   1
1   0   1
1   1   0

and say this data's real probability distribuiton is the blue one $P(Y|X)$ and our current model's (randomly initialized weights) probability disbribution is the red one $P(Y|X;\theta_{Random})$

enter image description here

Then the aim of Maximum Likelihood Estimation is to push the weights from randomly initialitzed weights to the real weights which approximately "describes" our real data distribution.

MLE = $ \{ P(Y|X;\theta_{Random})\rightarrow P(Y|X;\theta_{Real})\approx P(Y|X)\}$

Question 1: Is my thinking right; Is this the correct approach to describe MLE?

In gradient descent, to train a model we have to minimize a cost function but in MLE we are maximizing it. To solve this in here he talks about Negative Log Likelihood which is a negative added to the log converted MLE?

Question 2: How is adding a negative to log converted MLE maximises MLE?

My intiution tells me instead of learning the real distribution $P(Y|X;\theta_{Real})$ this should have pushed away the weights to some random distribution?

$\endgroup$
2
$\begingroup$

Maximum likelihood is a method for estimating parameters by maximizing the probability of the observed data. The main ingredients are:

  • The data: $D=(X,Y)$
  • The model parameters: $\theta$
  • The model that relates data to the parameters: $P(D|\theta)$ (which can be written differently, depending on the situation: $P(Y|X, \theta)$ is one of the possibilities.)

One would typically name the probability of observing data given parameters likelihood and write it as $L(\theta|D)=P(D|\theta)$. This is just a change of notation. One then maximizes the likelihood in respect to the values of the parameters: \begin{equation} \hat{\theta}=\mathrm{argmax}_\theta L(\theta|D). \end{equation}

Quite often the likelihood is a product of many identical functions for different data points. In this case it is mathematically more convenient to maximize its logarithm, i.e. the log-likelihood: $LL(\theta|D) = \log L(\theta|D)$. The likelihood and its logarithm has the same maximum, since logarithm is a monotonous function. Finally, maximizing a function is the same as minimizing its negative, therefore minimizing a negative log-likelihood is the same as maximizing the log-likelihood, which is the same as maximizing the likelihood. This is important since the optimization algorithms are often spelled explicitly only for minimization (or only for maximization).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.