2
$\begingroup$

I'm reading this page about cross-entropy loss and why it works as a maximum likelihood estimator.

He says:

Because we usually assume that our samples are independent and identically distributed, the likelihood over all of our examples decomposes into a product over the likelihoods of individual examples:

Then he gives an example: if our NN predicts (0.4, 0.1, 0.5) as the probabilities of the three classes, and the correct value is (1.0, 0.0, 0.0), then he says that the "likelihood" of that single example is just 0.4.

As I understand it, L(x, theta) = f(x, theta) for a single observation. How are we supposed to measure f(x, theta) when its value is itself a distribution? That is, what does it mean to have an outcome of (0.4, 0.1, 0.5) given a distribution of (1.0, 0.0, 0.0)?

Edit: if the question were "what is the probability of outcome 1 given distribution (0.4, 0.1, 0.5)", then there's no confusion. I've been interpreting it as "what is the probability of (0.4, 0.1, 0.5) given (1.0, 0.0, 0.0)" which seems awkward.

$\endgroup$
9
  • $\begingroup$ As this is a neural net, it doesn't really have a likelihood model on the output - What this 0.4 refers to, is most likely just the value for the cross-entropy-loss of that instance, which is - log(predicted probability for true class) $\endgroup$
    – Sam
    Aug 30, 2017 at 17:19
  • 1
    $\begingroup$ This looks perfectly ordinary to me, if we understand "predicts (0.4, 0.1, 0.5)" to mean that the estimated underlying distribution over three mutually exclusive classes assigns those probabilities to them. Since the likelihood--by definition--is the probability of the data and the data consist of the first class (which is what I presume "(1.0, 0.0, 0.0)" is intended to mean), then indeed its probability is $0.4$ according to the chosen model. See stats.stackexchange.com/questions/2641. $\endgroup$
    – whuber
    Aug 30, 2017 at 17:43
  • $\begingroup$ (@Sam) In this case, (0.4, 0.1, 0.5) is the output of a softmax layer (i.e., they are probabilities). We're then using it to calculate cross entropy loss. Does that help? $\endgroup$
    – monk
    Aug 30, 2017 at 17:44
  • 2
    $\begingroup$ The outcome (often called $x$) is class 1, if I'm reading your notation correctly. The probability law is $(0.4,0.1,0.5)$, often referred to as $\theta$. Thus the chance of this outcome for this particular probability law is $p_\theta(x)=0.4$. That is the likelihood of $\theta$ associated with the observation $x$. $\endgroup$
    – whuber
    Aug 30, 2017 at 18:05
  • 1
    $\begingroup$ Yes, I think you might be using some terms differently than intended. If I understand correctly what the network is doing, it is giving you its guess about a discrete distribution on three categories: that's $\theta$. It can be described by three non-negative numbers that sum to unity, as required of any probability distribution. An "outcome" is what we use to model the data ("our samples" or "examples" in your quotation). Its value $x$ can be any one of those three categories. You have encoded that value using three random variables. Their values are $(1,0,0)$, indicating category 1. $\endgroup$
    – whuber
    Aug 30, 2017 at 18:52

2 Answers 2

1
$\begingroup$

I'm not totally sure what you mean by "its value is itself a distribution," so let me say a few things and see if they help; feel free to ask more questions if not.

The network is predicting a discrete distribution over the three entries. Letting the predictive label be the random variable $\hat Y$ and naming its possible values $a$, $b$, and $c$, it says that $\Pr(\hat Y = a) = 0.4$, $\Pr(\hat Y = b) = 0.1$, and $\Pr(\hat Y = c) = 0.5$. Note that $\hat Y$ is a function of the network's parameters $\theta$ and the feature vector $x$: we can write it as $\hat Y_\theta(x)$ to denote its dependence on $\theta$ and $x$.

Now, we want to see if that predicted distribution is any good. We only have one data point to evaluate this with: the true observed value $y$, which in this case was observed as $a$. Taking a maximum-likelihood approach, we choose to evaluate the quality of a network $\theta$ by its likelihood: the probability of $a$ under the predictive distribution $\Pr(\hat Y_\theta(x) = \cdot)$, which we can evaluate as $P(\hat Y_\theta(x) = a) = 0.4$. (If the labels were continuous, then we'd use the probability density.)

Now, the network actually predicts one of these distributions for each of the possible inputs $x_i$; our measure of the overall quality of the network as a predictor is the sum of the likelihoods for each data sample. Because we assume these are iid, we get $$ \log \Pr\left( \big( \hat Y_\theta(x^{(i)}) \big)_{i=1}^n = \big( y^{(i)} \big)_{i=1}^n \right) = \log \prod_{i=1}^n \Pr\left( \hat Y_\theta(x^{(i)}) = y^{(i)} \right) = \sum_{i=1}^n \log \Pr\left( \hat Y_\theta(x^{(i)}) = y^{(i)} \right) .$$

The log-likelihood of a parameter value $\theta$ under the data $\{(x^{(i)}, y^{(i)}) \}_{i=1}^n$ is then $$ \ell(\theta) = \sum_{i=1}^n \log \Pr(\hat Y_\theta(x^{(i)}) = y^{(i)}) ,$$ which is what we want to maximize.


Compare to the case of finding the maximum-likelihood estimator for a series of biased coin flips. There the model is $\mathrm{Bernoulli}(\theta)$, i.e. $\Pr(\hat Y_\theta = H) = \theta$, $\Pr(\hat Y_\theta = T) = 1 - \theta$. The log-likelihood is $$\sum_{i=1}^n \begin{cases}\log(\theta) & y^{(i)} = H \\ \log(1 - \theta) & y^{(i)} = T\end{cases},$$ and we can estimate $\theta$ by maximizing $\ell(\theta)$ given the $y^{(i)}$.

The only difference in this case is that there are also feature vectors $x^{(i)}$, and we're maximzing the likelihood conditional on the $x^{(i)}$.

$\endgroup$
6
  • $\begingroup$ Thanks! I meant that (as you say) "the network is predicting a discrete distribution." When computing likelihoods, I'd only seen examples where the outcome is a single value. Suppose I have a coin where P(H)=p. We might ask "what is the likelihood of p given 'HTH'?" The question "what is the likelihood of p given 'H'?" is equivalent to "what is the probability of 'H' given p?" Here we have "what is the probability of (0.4, 0.1, 0.5) given (1.0, 0.0, 0.0)?" I don't know how to make sense of that. What you call "the natural way" sounds good to me, but is there theoretical justification? $\endgroup$
    – monk
    Aug 30, 2017 at 17:31
  • $\begingroup$ Also, when you say "the likelihood of $a$," you mean the probability of $a$, right? Likelihood should be of some parameter? $\endgroup$
    – monk
    Aug 30, 2017 at 17:34
  • $\begingroup$ Does my edit help? $\endgroup$
    – Danica
    Aug 30, 2017 at 17:41
  • $\begingroup$ I think the discussion on the question has pointed out where I went wrong: if the outcome is (1, 0, 0) and the distribution we're comparing with is (0.4, 0.1, 0.5) then I understand how to interpret the likelihood. But I've been assuming the that (0.4, 0.1, 0.5) is the outcome. $\endgroup$
    – monk
    Aug 30, 2017 at 18:19
  • 1
    $\begingroup$ It's all coming together now: my network is supposed to be outputting a model which achieves maximum likelihood given the labeled outcomes. That makes so much more sense than whatever I was thinking. Thanks! $\endgroup$
    – monk
    Aug 30, 2017 at 18:26
1
$\begingroup$

The question is confused: I was thinking of the predictions of the network as outcomes under a given model (represented by the labels). This is backward; the network is generating a model (parameterized by the network weights and inputs) under which the labels are the outcomes.

Therefore, the likelihood of a single example is equal to the probability of getting the labeled outcome (1.0, 0.0, 0.0) (referring to class 1) given the predicted distribution (0.4, 0.1, 0.5) and not vice-versa.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.