I've a set of 500+ observation (200+ feature vector dimension) of 7 classes and want improve my classification rate (with SVM or KNN).
To reduce the dimension and transform the feature matrix to a lower dimension (due to curse of dimensionality), I'm using LDA. It maps my high dimensional data to lower 6 dimensions. But with applying cross validated LDA it doesn't help and degrade the results dramatically.
When I even use leave one out (LOOCV) to calculate LDA projection matrix, it is calculated by holding out just one observation. My question is why even in this case the projection matrix ($W$) is so over-fitted and sensitive to cross validation? Intuitively I've hold out just one sample but it seems the projection matrix can't map the held out observation correctly.
I'm interested in two parts:
- The math behind such experiment.
- Some consideration or solution for better cross validated feature transform instead of LDA.
- based on @Andrew M, initial response, I've different number of observation per class. For example one class has example 120 observation while the other has only 40.