1
$\begingroup$

This is eye-opening, and the effect on KNN, for example, is easy to predict, but should the limitations of Euclidean distance in high dimension be a reason for concern in the very common application of multiple regression?

UPDATE: After all the comments it is clear that it is not the number of examples (subjects or observations or rows in the model matrix) that counts towards dimensionality: it is about the number of features (regressors, independent variables, columns in the model matrix) because no matter how many observations, in a model like $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 +\epsilon$, for example, you end up with a plane (2 dimensional).

enter image description here

Let me rephrase the question:

Why high dimensional vectors (many elements or rows of observations), $y\in \mathbb R^{\text{huge}}$, do not pose dimensionality issues, while many features (regressors or columns) do? Even if the desired model is low-dimensional, its calculation involves distances in high dimensional spaces.

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ Euclidean distance between what and what? // You may be interested in a question of mine from last year: stats.stackexchange.com/questions/492842/…. $\endgroup$
    – Dave
    Commented Feb 11, 2021 at 5:21
  • $\begingroup$ @Dave Have you made peace with the fact that your independent variable lives in $\mathbb R^n$, and the projection onto a hyperplane is what regression is about? The multiple observation of one dimension sounds at odds with the geometry. $\endgroup$
    – Antoni Parellada
    Commented Feb 11, 2021 at 5:52
  • $\begingroup$ I disagree with Stephan in the comments, and I still wonder why MSE is appropriate for large data sets. Is your question the same as mine, Euclidean distance between predictions and observed values? $\endgroup$
    – Dave
    Commented Feb 11, 2021 at 11:03
  • 1
    $\begingroup$ The concern is well-known and has a name: it goes under "multicollinearity." Except when data are collected according to designed experiments, observations in sufficiently high dimensions almost surely exhibit enough multicollinearity to be problematic. Search our site for threads on this topic, as well as threads on "variable selection," "model building," and "regularization." $\endgroup$
    – whuber ♦
    Commented Feb 11, 2021 at 19:24
  • 2
    $\begingroup$ @Dave The question of differences between a vector of values and a vector of predictions is purely two dimensional, according to Euclid, and so the dimensionality doesn't even arise as a concern. $\endgroup$
    – whuber ♦
    Commented Feb 12, 2021 at 14:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.