Why does the curse of dimensionality mimic multicollinearity, in the following sense..
- Consider the random vector $Y = [y_{1}, \dots, y_{4}]$ where each element is ~ Uniform (0,1).
- Take 10 samples of $Y$. Call this your dependent variable observations.
- Let the independent variable $X$ be 1 for the first 5 samples of $Y$, and 0 for the last 5 samples.
- Fit a multivariate regression for $Y | X$ (allow there to be a constant term in the regression).
Of course, the above fit succeeds. No problem.
Now let's increase the dimensionality.
- Now let $W = [w_{1}, \dots, w_{300}]$
- Take 10 samples of $W$, as above.
- Use the same $X$ as above.
- Fit a multivariate regression for $Y | X$.
The fit fails due to multicollinearity (says Matlab).
I can try scaling the observations by large positive constants - still fails.
Why does increasing the dimension mimic multicollinearity?
If I try a MANOVA test, I encounter same problem (of course, because ANOVA is regression).
Note: if I impose the constraint that the covariance matrix is diagonal, then it works. Why?
Matlab code for the above, for anybody interested:
Y = rand(300,10);
X = [ repmat([1 0], 5,1) ; repmat([1 1], 5,1) ];
mvregress(X,Y')
% generates error: "Covariance is not positive-definite."