Suppose I have a data-matrix $\bf X$, which has more features than samples ($p > n$).

I'd like to perform a Hotelling's T$^2$ test to determine whether or not to reject the null-hypothesis that the pop. mean is $\bf 0$.

I essentially face two problems, first, the covariance matrix is singular. Second, the F-statistic is computed as,


The problem of a singular covariane matrix I can correct either by regularizing it (adding a positive scalar to the diagonal) or taking the pseudo-inverse. But the fact that $p>n$ means that my F-statistic goes negative, which is nonsensical.

The obvious solution is to apply some sort of dimensionality reduction technique to $\bf X$ (e.g. PCA), but I'm still on the fence about whether or not this is appropriate.

Essentially, a transformation of the feature space would mean my hypothesis test is now testing a different hypothesis, i.e. whether my new set of features have mean $\bf 0$ in the population.

On the other hand, logically, I'm inclined to conclude that a rejection of the null hypothesis in a dimesionally reduced feature space should imply a rejection in the original feature space. That is to say, dimensionality reduction maintains specificity, while giving up some sensitivity. Is this the best I can hope for when faced with more features than samples?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.