1
$\begingroup$

I'm working with supervised classification of object-based satellite imageryand currently investigate different dimensionality reduction methods on their suitability to this application. As part of my work I'm also analysing the sensitivity to the training set size by visualising the learning curve (as implemented by scikit-learn).

I'm confused by the strange behaviour of Linear Discriminant Analysis (LDA) compared to that of the default Support Vector Machine (SVM) and to of Mutual-Information-based feature selection (MI) and Fisher's-criterion-based feature selection (See image below).

Learning Curves for different Classifiers

The graphs show the effect of the train-test-ratio (number of samples used for training/ number of samples used for testing) on the overall accuracy on the training data and the test data. The values are cross-validated using 6 stratified folds. The lines depicting the mean overall accuracy, the shaded areas indicating 3 standard deviations.

It is a multi-class classification problem with 10 classes with unbalanced classes ranging from ~100 to ~1000 samples per class. The initial feature set consisted of 712 features (depicted in 'Learning Curve - None'), LDA reduced this to 9 components, MI selected 185 features, F-Score selected 124 features.

The accuracy on the training data decreases with an increasing training set size. When both the train scores and validation scores start to converge it is usually an indicator that the classifier's accuracy cannot be improved further with larger training sets. See this reference from scikit-learn.

I'd appreciate any hints, on what could cause the weird behaviour of the LDA in the Train-test-ratio range from 0.025 (65 samples (1 in the least represented class)) over the minimum at 0.18 (~650 samples (~20 in the least represented class)) to 0.4 (~1000 samples (~40 in the least represented class)).

$\endgroup$
  • $\begingroup$ Please, explain your graphs a bit more. What is train-test ratio precisely, what shaded error areas represent, why do accuracy with train data also change, what is the number of features. $\endgroup$ – ttnphns Oct 17 '18 at 8:29
  • $\begingroup$ I've tried to describe it in more detail. I hope it helps to clarify. $\endgroup$ – Oliver Stromann Oct 17 '18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.