Multivariate analysis of variance I am doing multivariate analysis of variance for a dataset composed of m observation in n dimensions (n>>m, that is n>10000 and m=20).
the problem is that it looks like that I have some variables are linearly dependent as  I got the following error message:

The within-group sum of squares and cross products matrix is singular. 

So, what is the best solution for such problem? (applying PCA and SVD might not be suitable as the covariance matrix will be 10000x10000 and will be difficult to process it)?
 A: The best solution depends on the exact nature of the problem. 
One literature you might want to look into is the one around "age-period-cohort models". We might expect that someone's age, the era in which people grow up (the cohort), and the period in which the question was asked all have their own influence on a person behavior, but they are linearly dependent: year of birth (cohort) = period - age. There is a whole set of models that put different constraints on the different parameters in order to try to identify the different effects. These models are often refered to as "age-period-cohort models".
A: Why do you think MANOVA will answer your question? It will give you the (linear) combination of the m variables that maximize the difference between the centroids (k = 2, in your case) assuming that dispersion is the same (and that everything is multi-normal...).
So that seems almost the opposite of your interest of testing for differences in dispersion (as you commented below your question). By the way, as far as I know you need m (per group) > n to get the eigenvalues and eigenvectors. Without m >> n you won't have enough power for significance tests.
If what you want to resume what all those variables have to say about describing your observations, then PCA may be useful as an indirect method (i.e., first resume those variables in independent components that maximize the explanation of samples heterogeneity, then compare groups based on those components) and linear correlation among variables is not a problem there (is what the components are all about). The particular solution (e.g., important variables on the first component) will be very unstable, anyway, because n << m. 
