1
$\begingroup$

I'm performing a binary sentiment classification (positive/negative) based on a Naive Bayes classificator and a SVM. To select top k features I use the MRMR algorithm. The model is trained using a 10-fold cross validation and then tested on unseen data.

As you can see, I plotted the top k features from 25 to 350 and the accuracy as well as the f-measure since the algorithms have trouble to predict positive classes. Who can help me to understand them?

Why do they converge and diverge? And what is the "best" model (main criterion is f-measure) and the best size of k according to the graphs?

enter image description here

enter image description here

enter image description here

enter image description here

$\endgroup$
  • $\begingroup$ Nobody an idea? Up to a certain number of features the trainingdata is constantly overperforming the testingdata. Is it because the number of features/featureset is just appropriate for the trainingdata? And after a while the testingdata can be explained by the featureset too? $\endgroup$ – user18075 Feb 11 '13 at 10:29
  • $\begingroup$ How large are your two data sets? How did you determine the performance on the training data? Is it determined by looking at the results of the classifier trained on the entire training data or is based on cross validation and/or bootstrapping? If cross validation is it a nested one, which contains the cross validation used to tune your model parameters? $\endgroup$ – Erik Feb 11 '13 at 12:24
  • $\begingroup$ The dataset consists of 360 positive and 360 negative examples. The performance of the training data is determined by the avarage values of a 10-fold cross validation. The training is solely done on training data (0.7 percent of all 720 examples [stratified]). The testing is performed on completly unseen data (0.3 percent of all 720 examples [stratified]). Model parameters have not been adapted (default settings). $\endgroup$ – user18075 Feb 11 '13 at 15:16
  • $\begingroup$ Just to clarify, the cross validation that gives the performance is the same one where the (for example) SVM parameters are tuned? $\endgroup$ – Erik Feb 11 '13 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.