I am trying to perform a classification between 20K objects to 40 classes.
The features of the classification are text based, per each objects i have its raw text.
In addition i have a ~300 training sample, roughly 8 samples per class.
Ive tried using the following method:
1) Vectorize the data - using tf_idf.
2) Decomposing the vectors - using NMF.
3) Training all sorts of classifiers on decomposed vectors.
My problem are the big differences between training and test sets prediction accuracy, even after selecting only few latent variables(post decomposition).
- If i use decomposition and selecting only top latent variables -
train prediction accuracy(PA) is around the 30% and test's is at 10%.
- If i dont use decomposition and train the classifier using the vectors (tried vectors shapes sizes from 1000 to 10000 by capping max_features in the vectorizer) -
train PA is ~100% and test PA is ~40%.
Ive grid-searched different vectorizing, decomposition and classifiers hyper params, and kept getting the same difference in prediction accuracy.
1) Is it an overfit problem?
2) If so, how come its still happening even when choosing only few features for classification?
3) Is my method well suited for this problem? are there any other options i should consider?