The likelihood:

In statistics, the likelihood function (often simply called the likelihood) measures the goodness of fit of a statistical model to a sample of data for given values of the unknown parameters.

I don't understand this likelihood function:

$$LH = \prod q_i^{Np_i} \hspace{1em} (1)$$

$q_i$ are probabilities estimated with some model, $p_i$ the empirical probabilities (from training set). $N$ is confusing but it's just the number of times we ask for $i$.

Now, How exactly is this supposed to be used? Say we get $q_i$ using some function, say it's $0.88$ for cats, $0.12$ for non-cats,

$$LH = 0.88^{p_{cat}}\hspace{1em}(2)$$

And if ask also for 2 cats and 1 non cats:

$$LH(cat,nocat) = 0.88^{2*p_{cat}}*0.12^{1*p_{nocat}} \hspace{1em}$$

But how exactly is the rest calculated and, more importantly, interpreted?


2 Answers 2


Likelihood gives "the probability of getting data".

Let's talk about the classical coin flip problem. Suppose I have two observations, Head and Tail, and want to estimated the coin parameter $\theta$, the chance of getting head.

If we are setting the $\theta = 0.5$, the the likelihood (probability of getting data) is $0.5 \times0.5 = 0.25$.

If we are setting the $\theta = 0.01$, the the likelihood is $0.01 \times 0.99=0.0099$.

So, we may choose $\theta=0.5$. Because it is getting a larger likelihood.

In the model fitting game, we are tuning parameters to see if we can get a larger likelihood and a larger likelihood can get we are getting a better fitting.

  • $\begingroup$ Yes but how about the formula I'm posting? $\endgroup$
    – Minsky
    Commented Jan 27, 2021 at 8:23

That particular example that you gave is such one that relates the maximization of the log Likelihood, $l(q)$ with the Cross Entropy, $H(q,p)$.

Let's formulate the example that you posted. We have to possible events $Cat$ and $NonCat$, each with unknown probability $q_{1}$ and $q_{2}=1-q_{1}$.

We run an experiment with $N$ independent trials and compute the likelihood of the experiment which is

$$L(q)=q_{1}^{Np_{1}}\times q_{2}^{Np_{2}} \tag 1$$

essentially the $Np_{i},$ because the $p_{i}$ is the proportion of the event $i$ over the trials, is equal to $n_{i}$ individuals that correspond to the class $i$.

If you try and maximize the $(1)$, you will take maximum likelihood estimation for $q_{1}$ and $q_{2},$ $\frac{n_{1}}{N}$ and $\frac{n_{2}}{N}$ respectively, which can be interpreted as the estimations that minimized the dissimilarity between the $q_{1},q_{2}$ and $p_{1},p_{2}$ and the estimations that make the observed (training) data more probable.

However, an equivalent way of viewing this problem is throughtout the Cross Entropy. We have the logLikelihood divided by the $N$


So, either you maximize the $\frac{l(q)}{N}$ where $N$ is just a constant or maximize the negative Cross Entropy $-H(q,p)$ which is equivalent to minimizing the actual Cross Entropy $H(q,p).$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.