0
$\begingroup$

Let’s set up a supervised learning problem with $p$ predictors and $n$ observations. The response variable is univariate. The problem can be regression or classification, though I think a classification problem introduces additional complexity if there are more than two categories.

Euclidean distance between two vectors, which is quite related to square loss where the two vectors are the $n$-dimensional prediction vector and the $n$-dimensional vector of observed responses, has problems when the dimension is high (whatever “high” means).

Why is Euclidean distance not a good metric in high dimensions?

Nonetheless, square loss is popular, even when there are hundreds of observations and therefore dimensions (which should be enough to trigger some of the bizarre behavior of the Euclidean norm).

Yes, if we use square loss for OLS regression under the Gauss-Markov conditions, we get the BLUE. Yes, we can do all sorts of inference on the parameters. However, I am thinking of a pure prediction problem, perhaps a complicated neural network. In that kind of prediction problem, the inference and interpretation are much less important than the predictive accuracy.

So why use square loss when there are many observations?

EDIT

I say that the observations $y$ and predictions $\hat y$ are vectors, even for a univariate response, because they must be for the following to make sense as linear transformations.

$$ \hat\beta_{ols}=(X^TX)^{-1}X^Ty\\ \hat y=X(X^TX)^{-1}X^Ty $$

The upper equation represents a linear transformation of a vector $y\in\mathbb R^n$ to $\hat\beta_{ols}\in\mathbb R^p$, and the bottom represents a linear transformation from $\mathbb R^n\rightarrow\mathbb R^n$.

$\endgroup$
11
  • 2
    $\begingroup$ If your response variable is univariate, then you have $n$ points in a single dimension - not in $n$ dimensions. Are you thinking of multivariate prediction problems? $\endgroup$ Oct 20, 2020 at 14:09
  • $\begingroup$ @StephanKolassa I'm considering $y\in\mathbb{R}^n$ and $\hat{y}\in\mathbb{R}^n$. Square loss is then $d_{L2}(y,\hat{y})$. I say this is reasonable, as we often write something like $\hat{\beta} = (X^TX)^{-1}X^Ty$, where we treat $y$ as being in $\mathbb{R}^n$. $\endgroup$
    – Dave
    Oct 20, 2020 at 14:48
  • 3
    $\begingroup$ Well, but then my initial comment applies, and you have only a single dimension. I think I'm confused. Can you perhaps give a concrete example of what you are trying to do? $\endgroup$ Oct 20, 2020 at 15:06
  • 3
    $\begingroup$ OK. So you have a single dimension and a lot of observations in this single dimension. The Euclidean metric has no problems in such a setting that I am aware of. Can you explain what problems you see? $\endgroup$ Oct 20, 2020 at 15:58
  • 4
    $\begingroup$ You have 20 million observations in a single dimension, speed. A multivariate analysis would predict, e.g., a triple (speed, height, weight), and then you would measure 20 million individuals on these three dimensions. But three dimensions is still not "high-dimensional". Can you explain what problems you see? $\endgroup$ Oct 20, 2020 at 16:03

2 Answers 2

0
$\begingroup$

I wasn’t thinking like a statistician when I posted this two years ago.

Under an assumption of a Gaussian error term, minimizing square loss is equivalent to maximum likelihood estimation.

When we find the parameter estimates that minimize square loss, there is a sense in which those are the most likely parameter values, given the data.

I find this answer somewhat unsatisfying because I wanted to think purely in terms of doing predictions, but perhaps the relationship to maximum likelihood estimation and the tradition of minimizing square loss are the best I can do (and perhaps finding the “most likely” estimates really is a reasonable way to assess predictive accuracy).

$\endgroup$
0
$\begingroup$

How much data you have is irrelevant to the choice of loss function. If you have more data, lucky you. Loss is a part of your statistical model, your assumptions about the data as it corresponds to the likelihood function. Loss is also about characteristics of the data, for example you may want it to be more robust against some kinds of outliers.

As about Euclidean distance, the problem relates to high dimensional data, i.e. many columns, not many rows (“big data”). Usually when minimizing loss you look at average loss across the samples, so it doesn’t really matter how many samples you have.

Why squared loss? It is one of the simplest ones, it is mathematically convenient (corresponds to Gaussian likelihood, has simple derivative), and for historical reasons. See Why is the squared difference so commonly used? and What makes mean square error so good? for more details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.