Square loss for "big data"

Question

Let’s set up a supervised learning problem with $p$ predictors and $n$ observations. The response variable is univariate. The problem can be regression or classification, though I think a classification problem introduces additional complexity if there are more than two categories.

Euclidean distance between two vectors, which is quite related to square loss where the two vectors are the $n$-dimensional prediction vector and the $n$-dimensional vector of observed responses, has problems when the dimension is high (whatever “high” means).

Why is Euclidean distance not a good metric in high dimensions?

Nonetheless, square loss is popular, even when there are hundreds of observations and therefore dimensions (which should be enough to trigger some of the bizarre behavior of the Euclidean norm).

Yes, if we use square loss for OLS regression under the Gauss-Markov conditions, we get the BLUE. Yes, we can do all sorts of inference on the parameters. However, I am thinking of a pure prediction problem, perhaps a complicated neural network. In that kind of prediction problem, the inference and interpretation are much less important than the predictive accuracy.

So why use square loss when there are many observations?

EDIT

I say that the observations $y$ and predictions $\hat y$ are vectors, even for a univariate response, because they must be for the following to make sense as linear transformations.

$$ \hat\beta_{ols}=(X^TX)^{-1}X^Ty\\ \hat y=X(X^TX)^{-1}X^Ty $$

The upper equation represents a linear transformation of a vector $y\in\mathbb R^n$ to $\hat\beta_{ols}\in\mathbb R^p$, and the bottom represents a linear transformation from $\mathbb R^n\rightarrow\mathbb R^n$.

Answer 1

I wasn’t thinking like a statistician when I posted this two years ago.

Under an assumption of a Gaussian error term, minimizing square loss is equivalent to maximum likelihood estimation.

When we find the parameter estimates that minimize square loss, there is a sense in which those are the most likely parameter values, given the data.

I find this answer somewhat unsatisfying because I wanted to think purely in terms of doing predictions, but perhaps the relationship to maximum likelihood estimation and the tradition of minimizing square loss are the best I can do (and perhaps finding the “most likely” estimates really is a reasonable way to assess predictive accuracy).

Answer 2

How much data you have is irrelevant to the choice of loss function. If you have more data, lucky you. Loss is a part of your statistical model, your assumptions about the data as it corresponds to the likelihood function. Loss is also about characteristics of the data, for example you may want it to be more robust against some kinds of outliers.

As about Euclidean distance, the problem relates to high dimensional data, i.e. many columns, not many rows (“big data”). Usually when minimizing loss you look at average loss across the samples, so it doesn’t really matter how many samples you have.

Why squared loss? It is one of the simplest ones, it is mathematically convenient (corresponds to Gaussian likelihood, has simple derivative), and for historical reasons. See Why is the squared difference so commonly used? and What makes mean square error so good? for more details.