4
$\begingroup$

I am new to statistics and currently got a dataset which contains $80$ dependent variables and $1$ independent variable with $2$ groups. MANOVA reports a $p$-value of $> 0.6$ on this dataset. But when I use linear discriminant analysis (LDA), two groups can be separated perfectly (almost $100\%$).

Is this result possible?

What I understand for MANOVA is that larger $p$-value means less difference between two groups. Then how can the groups be separated perfectly if there is no difference between them?

$\endgroup$
2
  • 1
    $\begingroup$ P-value depends on sample size (which might be low in your case) as well as degrees of freedom (which is 80 in your case), so it can be high (nonsignificant) even with well-separable groups. (Note btw that since you have just 2 groups, your MANOVA and discriminant analysis are equivalent to linear regression of your IV by the 80 DVs: R-square = Pillai's trace.) $\endgroup$ – ttnphns Oct 22 '14 at 7:01
  • $\begingroup$ How many data points (samples) do you have in each group? $\endgroup$ – amoeba Oct 22 '14 at 11:06
5
$\begingroup$

This is an excellent question because it touches on so many important concepts. The short answer is: Yes, this is possible, and can happen if your sample size is low.


Let us make the apparent contradiction a bit more precise. The MANOVA tests whether your data could have been observed if in reality there were no difference between the two groups (that is the null hypothesis). Your $p$-value $p=0.6$ is telling you that the answer is: yes, it easily could. At the same time, LDA results in a [almost] perfect separation between the two groups. So is it possible that in reality there is no difference between the groups but the actual data appears to be perfectly separable?

We can use a simple simulation to check. For various values of the sample size $N$ I generated random data from the standard normal distribution in the $80$-dimensional space $\mathbb R^{80}$ and assigned one half of the points to group $\#1$ and another half to group $\#2$. Both groups are therefore sampled from identical distributions, with true means at zero. For any value of $N$, MANOVA usually reports a high non-significant $p$-value, as expected. But let us look at LDA.

First of all, note that if $N<80$ then the groups can always be separated perfectly. Think e.g. of two points in 2D or of three points in 3D: however you assign them to two groups, one can always linearly separate them. On the other hand, if $N$ is huge, e.g. $N=1\:000\:000$, then it is intuitively clear that the two massive clouds of points (group $\#1$ and group $\#2$) will be entirely overlapping, resulting in no separation. But it might be surprising to see how slow the apparent separation is decreasing with increasing $N$:

LDA overfitting

The blue line shows mean values of classification accuracy for $N$ between $100$ and $1000$ (mean over $100$ repeated simulations), and the shading shows two standard deviations. At $N=100$ separation is almost perfect. At $N=200$ separation is around $80\%$. At $N=500$ it is still over $65\%$. One needs to get to $N>100\:000$ to get below $51\%$.

This effect is known as overfitting. Your $80$-dimensional space is large. LDA is looking for an axis with the best separability between groups, and when the sample size is not big enough, it can usually find some axis that by chance happens to yield good separability. That is why one should better use cross-validation to assess the performance of a classifier: if we used it here, then cross-validated classification accuracy would always be around $50\%$, as it should be.

Technically, overfitting happens because within-class covariance matrix cannot be reliably estimated with small $N$ (and so the sample covariance matrix in the example above will have some very small eigenvalues, instead of them all being equal). You might be interested in reading more in my answers here:

$\endgroup$
4
  • $\begingroup$ (+1): you are on a roll today! $\endgroup$ – user603 Oct 23 '14 at 10:53
  • $\begingroup$ @amoeba Thank you very much for this comprehensively answers. Yes my sample size is small, I thought it was something related to the sample size but I just don't know how to describe it. $\endgroup$ – Adrian Oct 28 '14 at 4:56
  • $\begingroup$ BTW, looks like the title of my question has changed automatically. I can't find it at first when I search with my original title (I asked as a guest). I am just curious if there is an administrator or someone who modifies people's questions? $\endgroup$ – Adrian Oct 28 '14 at 5:05
  • $\begingroup$ @Adrian: On StackExchange websites everybody can suggest edits to any question or answer, and users with high reputation can accept or reject suggested edits (they can also directly edit themselves). Questions are routinely edited to improve formatting, wording, tags, title, etc. It is particularly important that the titles are concise and clear. In this case it was me who edited your question (you can see it yourself: all the edits are saved and logged); I hope it was an improvement. If you disagree, you can always roll back. $\endgroup$ – amoeba Oct 28 '14 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.