1
$\begingroup$

Suppose I have a data-matrix $\bf X$, which has more features than samples ($p > n$).

I'd like to perform a Hotelling's T$^2$ test to determine whether or not to reject the null-hypothesis that the pop. mean is $\bf 0$.

I essentially face two problems, first, the covariance matrix is singular. Second, the F-statistic is computed as,

$$F=\frac{n-p}{p(n-1)}T^2.$$

The problem of a singular covariane matrix I can correct either by regularizing it (adding a positive scalar to the diagonal) or taking the pseudo-inverse. But the fact that $p>n$ means that my F-statistic goes negative, which is nonsensical.

The obvious solution is to apply some sort of dimensionality reduction technique to $\bf X$ (e.g. PCA), but I'm still on the fence about whether or not this is appropriate.

Essentially, a transformation of the feature space would mean my hypothesis test is now testing a different hypothesis, i.e. whether my new set of features have mean $\bf 0$ in the population.

On the other hand, logically, I'm inclined to conclude that a rejection of the null hypothesis in a dimesionally reduced feature space should imply a rejection in the original feature space. That is to say, dimensionality reduction maintains specificity, while giving up some sensitivity. Is this the best I can hope for when faced with more features than samples?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.