When is there a difference between a normal likelihood loss and a least squares loss?

Question

My understanding is that if the errors follow a normal distribution, then using a maximum likelihood loss or a least squares loss to train a model amounts to the same thing.

However, I am looking at some code for a neural network time series prediction model, and it includes as an input argument the option to specify whether a squared loss or a normal likelihood loss is used, along with a comment that if normal likelihood loss is specified, then the covariance term is trained as well.

3 things I don't get about this:

1. Don't a likelihood loss and a squared loss amount to the same thing if the distribution is normal (see here)?
2. What is a covariance term in the case of time series?
3. What does "training the covariance term" mean exactly in the context of neural networks? That both the loss and the covariance are minimized?

Answer 1

Looking at the link you provided gets you the definition of normal loss as:

$$\frac{-n}{2} \log(\sigma^{2}) - \frac{1}{2 \sigma^{2}} \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2}$$

If you assume that $\sigma^2$ (the variance) is fixed this reduces to a square error loss. However if you want to perform inference on $\sigma^2$ you'll optimize the loss function with respect to both $\sigma^2$ and $\beta$, learning values for both.

In the code you're looking at it sounds like you have the option to ignore $\sigma^2$ or attempt to learn a value. In either case the model ought to make the same point predictions, but if you learn $\sigma^2$ you'll also model the variability of the outputs.

Answer 2

Neural networks are typically positioned as classifiers -- so your final output is Yes/No -- or possibly a class with more than two options. But let's suppose that it's only a binary class. In that case, you are modelling a lot of Bernouilli trials. The maximum likelihood will contain terms like $y\log(p) + (1-y)\log(1-p)$, where $y$ is 0 or 1, and $p$ is the probability of a success for that individual.

On the other hand, you could simply slap a bunch of squared error losses together, like $(y-\hat{y})^2$, where $\hat{y}$ would be an estimated probability of success.

Back in the day, when I learned statistics, before logistic regression was invented, we regressed variables against binary outcomes by using 0 and 1 as responses and blasting a regression line at the problem. In other words, we used squared error loss. This isn't such a bad thing when the probabilities are in the mid range. Things get bad when the probabilities are close to 0 or 1.

But to answer your question, you are not dealing with normal errors here and the maximum likelihood differs from squared error loss.

Answer 3

One case is more general than the other.

1. The log likelihood for multivariate normal distributed errors is related to a term $$-\frac{1}{2} \log \left(\vert \mathbf{\Sigma} \vert\right) {-\frac{1}{2} \mathbf{\epsilon}^T \mathbf{\Sigma}^{-1}\mathbf{\epsilon}} = -\frac{1}{2} \log \left(\vert \mathbf{\Sigma} \vert \right){-\frac{1}{2}\sum s_{ij}\epsilon_i\epsilon_j}$$ where $\mathbf{\Sigma}$ is the covariance matrix, $s_{ij}$ are the entries of it's inverse, and $\mathbf{\epsilon}$ is a $1$ x $n$ vector with the residuals/errors.

   note that this is not a simple sum of squares $\epsilon_i\epsilon_i=\epsilon_i^2$ but also contains mixed product terms $\epsilon_i\epsilon_j$

2. If $\mathbf{\Sigma}$ is diagonal ($\mathbf{\Sigma}_{d}=0$ for non-diagonal entries) then this reduces to a simple sum of squares. $${ -\frac{1}{2} \log \left(\vert \mathbf{\Sigma_d} \vert \right)-\frac{1}{2} \mathbf{\epsilon}^T \mathbf{\Sigma}_d^{-1}\mathbf{\epsilon}} = -\frac{1}{2} \sum \log \left(\sigma_i^2 \right) {-\frac{1}{2}\sum\frac{\epsilon_i^2}{\sigma_i^2}}$$

3. or even more simpler when $\mathbf{\Sigma} = \sigma^2\mathbf{I}$ $$ -\frac{1}{2} \log \left(\vert \sigma^2\mathbf{I} \vert \right) {-\frac{1}{2} \mathbf{\epsilon}^T (\sigma^2\mathbf{I})^{-1}\mathbf{\epsilon}} = -\frac{n}{2} \log \left(\sigma^2 \right) {-\frac{1}{2\sigma^2}\sum \epsilon_i^2}$$ which is maximized by changing $\beta$ and $\sigma$ and independent from $\sigma$ the maximum occurs when $\sum \epsilon_i^2= \sum (y_i-\mathbf{x_i}\mathbf{\beta})^2$ is minimized.

---

So only for the second and third case do you have a plain sum of squares expression. In the first case there are also cross terms, and more importantly there are more parameters to estimate (ie. all the correlation terms in the matrix $\mathbf{\Sigma}$ ) That is what is meant in the statement

if normal likelihood loss is specified, then the covariance term is trained as well.

If you wish you could transform/rotate the variables and then the first case could be expressed as the second case with a sum of squares. But then it becomes a bit a semantic issue. You would still have to compute more parameters (for the transformation) and you are not using an ordinary least squares loss function.

---

Q1: The multivariate normal likelihood is more generally a second order polynomial of the error/residual terms (with terms $\epsilon_i\epsilon_j$ for which not necessary $i=j$). Only in the special cases (independent errors) does this polynomial only contain the square terms (with $\epsilon_i\epsilon_j$ and only $i=j$)

Q2: The covariance term is the covariance term in the multivariate normal distribution that is used to express the errors. The covariance term allows to model errors that are dependent/correlated. In the case of time series this is probably correlation between errors of nearby time points, since it would be difficult (if possible at all) to estimate the entire covariance matrix $\mathbf{\Sigma}$ when there are no imposed restrictions. (a common model where error terms are considered to be correlated is a mixed effects model, see for an intuitive example https://stats.stackexchange.com/a/330097/164061)

Q3: Training the covariance terms means that the covariance terms are estimated in addition to the parameters of interest (e.g. a sample mean), as part of finding the minimum of the likelihood function. The covariance terms are nuisance parameters.

---

^{It may also be that a single measurement $y_i$ is a vector and that the errors in that vector are considered to be correlated.}