My understanding is that if the errors follow a normal distribution, then using a maximum likelihood loss or a least squares loss to train a model amounts to the same thing.

However, I am looking at some code for a neural network time series prediction model, and it includes as an input argument the option to specify whether a squared loss or a normal likelihood loss is used, along with a comment that if normal likelihood loss is specified, then the covariance term is trained as well.

3 things I don't get about this:

  1. Don't a likelihood loss and a squared loss amount to the same thing if the distribution is normal (see here)?
  2. What is a covariance term in the case of time series?
  3. What does "training the covariance term" mean exactly in the context of neural networks? That both the loss and the covariance are minimized?
  • 2
    $\begingroup$ As the mention of 'covariance' suggests, consider likelihood for a multivariate normal with unknown $\Sigma$ $\endgroup$ – Glen_b -Reinstate Monica Jun 21 '18 at 18:24
  • 2
    $\begingroup$ Please don't attempt to call a specific user to answer questions. In any case, it wouldn't work - unless Sycorax has participated in this post in some way, they will not be notified of your message. $\endgroup$ – Glen_b -Reinstate Monica Jun 22 '18 at 0:50

Looking at the link you provided gets you the definition of normal loss as:

$$\frac{-n}{2} \log(\sigma^{2}) - \frac{1}{2 \sigma^{2}} \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2}$$

If you assume that $\sigma^2$ (the variance) is fixed this reduces to a square error loss. However if you want to perform inference on $\sigma^2$ you'll optimize the loss function with respect to both $\sigma^2$ and $\beta$, learning values for both.

In the code you're looking at it sounds like you have the option to ignore $\sigma^2$ or attempt to learn a value. In either case the model ought to make the same point predictions, but if you learn $\sigma^2$ you'll also model the variability of the outputs.

  • $\begingroup$ I'm not sure that's relevant to the issue at hand - namely the loss function to be used in a neural network. $\endgroup$ – Placidia Jun 21 '18 at 18:34
  • $\begingroup$ Even though the question is asked in the context of a neural network model I don't think the answers depend on the model being an NN. As long as the error distribution is assumed to be Gaussian there is a connection between the likelihood loss and the square error loss, and "training the covariance" has a specific meaning. $\endgroup$ – Max S. Jun 21 '18 at 18:49

One case is more general than the other.

  1. The log likelihood for multivariate normal distributed errors is related to a term $$-\frac{1}{2} \log \left(\vert \mathbf{\Sigma} \vert\right) {-\frac{1}{2} \mathbf{\epsilon}^T \mathbf{\Sigma}^{-1}\mathbf{\epsilon}} = -\frac{1}{2} \log \left(\vert \mathbf{\Sigma} \vert \right){-\frac{1}{2}\sum s_{ij}\epsilon_i\epsilon_j}$$ where $\mathbf{\Sigma}$ is the covariance matrix, $s_{ij}$ are the entries of it's inverse, and $\mathbf{\epsilon}$ is a $1$ x $n$ vector with the residuals/errors.

    note that this is not a simple sum of squares $\epsilon_i\epsilon_i=\epsilon_i^2$ but also contains mixed product terms $\epsilon_i\epsilon_j$

  2. If $\mathbf{\Sigma}$ is diagonal ($\mathbf{\Sigma}_{d}=0$ for non-diagonal entries) then this reduces to a simple sum of squares. $${ -\frac{1}{2} \log \left(\vert \mathbf{\Sigma_d} \vert \right)-\frac{1}{2} \mathbf{\epsilon}^T \mathbf{\Sigma}_d^{-1}\mathbf{\epsilon}} = -\frac{1}{2} \sum \log \left(\sigma_i^2 \right) {-\frac{1}{2}\sum\frac{\epsilon_i^2}{\sigma_i^2}}$$

  3. or even more simpler when $\mathbf{\Sigma} = \sigma^2\mathbf{I}$ $$ -\frac{1}{2} \log \left(\vert \sigma^2\mathbf{I} \vert \right) {-\frac{1}{2} \mathbf{\epsilon}^T (\sigma^2\mathbf{I})^{-1}\mathbf{\epsilon}} = -\frac{n}{2} \log \left(\sigma^2 \right) {-\frac{1}{2\sigma^2}\sum \epsilon_i^2}$$ which is maximized by changing $\beta$ and $\sigma$ and independent from $\sigma$ the maximum occurs when $\sum \epsilon_i^2= \sum (y_i-\mathbf{x_i}\mathbf{\beta})^2$ is minimized.

So only for the second and third case do you have a plain sum of squares expression. In the first case there are also cross terms, and more importantly there are more parameters to estimate (ie. all the correlation terms in the matrix $\mathbf{\Sigma}$ ) That is what is meant in the statement

if normal likelihood loss is specified, then the covariance term is trained as well.

If you wish you could transform/rotate the variables and then the first case could be expressed as the second case with a sum of squares. But then it becomes a bit a semantic issue. You would still have to compute more parameters (for the transformation) and you are not using an ordinary least squares loss function.

Q1: The multivariate normal likelihood is more generally a second order polynomial of the error/residual terms (with terms $\epsilon_i\epsilon_j$ for which not necessary $i=j$). Only in the special cases (independent errors) does this polynomial only contain the square terms (with $\epsilon_i\epsilon_j$ and only $i=j$)

Q2: The covariance term is the covariance term in the multivariate normal distribution that is used to express the errors. The covariance term allows to model errors that are dependent/correlated. In the case of time series this is probably correlation between errors of nearby time points, since it would be difficult (if possible at all) to estimate the entire covariance matrix $\mathbf{\Sigma}$ when there are no imposed restrictions. (a common model where error terms are considered to be correlated is a mixed effects model, see for an intuitive example https://stats.stackexchange.com/a/330097/164061)

Q3: Training the covariance terms means that the covariance terms are estimated in addition to the parameters of interest (e.g. a sample mean), as part of finding the minimum of the likelihood function. The covariance terms are nuisance parameters.

It may also be that a single measurement $y_i$ is a vector and that the errors in that vector are considered to be correlated.


Neural networks are typically positioned as classifiers -- so your final output is Yes/No -- or possibly a class with more than two options. But let's suppose that it's only a binary class. In that case, you are modelling a lot of Bernouilli trials. The maximum likelihood will contain terms like $y\log(p) + (1-y)\log(1-p)$, where $y$ is 0 or 1, and $p$ is the probability of a success for that individual.

On the other hand, you could simply slap a bunch of squared error losses together, like $(y-\hat{y})^2$, where $\hat{y}$ would be an estimated probability of success.

Back in the day, when I learned statistics, before logistic regression was invented, we regressed variables against binary outcomes by using 0 and 1 as responses and blasting a regression line at the problem. In other words, we used squared error loss. This isn't such a bad thing when the probabilities are in the mid range. Things get bad when the probabilities are close to 0 or 1.

But to answer your question, you are not dealing with normal errors here and the maximum likelihood differs from squared error loss.

  • 1
    $\begingroup$ In this case the neural network is definitely being used as a regressor not a classifier, so the corss entropy reasoning doesn't hold I'm pretty sure. $\endgroup$ – Reinstate Monica Jun 21 '18 at 17:25
  • $\begingroup$ It's still not a pure "normal" estimation. The neural network itself basically chains together a bunch of logistic models, or some other non-linear functions of the input variables. Otherwise, the whole thing would collapse to a basic multiple regression. Why not try and write down the likelihood function for the model you have -- then you will see that it isn't the same as squared error loss. $\endgroup$ – Placidia Jun 21 '18 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.