I'm working with supervised classification of object-based satellite imageryand currently investigate different dimensionality reduction methods on their suitability to this application. As part of my work I'm also analysing the sensitivity to the training set size by visualising the learning curve (as implemented by scikit-learn).
I'm confused by the strange behaviour of Linear Discriminant Analysis (LDA) compared to that of the default Support Vector Machine (SVM) and to of Mutual-Information-based feature selection (MI) and Fisher's-criterion-based feature selection (See image below).
The graphs show the effect of the train-test-ratio (number of samples used for training/ number of samples used for testing) on the overall accuracy on the training data and the test data. The values are cross-validated using 6 stratified folds. The lines depicting the mean overall accuracy, the shaded areas indicating 3 standard deviations.
It is a multi-class classification problem with 10 classes with unbalanced classes ranging from ~100 to ~1000 samples per class. The initial feature set consisted of 712 features (depicted in 'Learning Curve - None'), LDA reduced this to 9 components, MI selected 185 features, F-Score selected 124 features.
The accuracy on the training data decreases with an increasing training set size. When both the train scores and validation scores start to converge it is usually an indicator that the classifier's accuracy cannot be improved further with larger training sets. See this reference from scikit-learn.
I'd appreciate any hints, on what could cause the weird behaviour of the LDA in the Train-test-ratio range from 0.025 (65 samples (1 in the least represented class)) over the minimum at 0.18 (~650 samples (~20 in the least represented class)) to 0.4 (~1000 samples (~40 in the least represented class)).