I need to compare SVM and NB about sentiment classification by evaluating accuracy, precision and recall measures. I have 1500 manually classified documents, and I would know which is the best way to compare these two algorithms, also increasing training set size from 100 to 1000 documents. I'm using scikit-learn and there are different methods, like KFold, ShuffleSplit, StratifiedKFold, StratifiedShuffleSplit. Which is the right one for my needs?
My approach is to use ShuffleSplit, using a fixed random seed, to first shuffle docs and then select 10 iterations of X training documents and 500 test documents, where X varies from 100 to 1000.
So, I have:
- cycle 1) 10 iterations with training set of 100 docs and test set of 500 docs;
- cycle 2) 10 iterations with training set of 100 docs and test set of 500 docs;
- ...
- cycle 10) 10 iterations with training set of 1000 docs and test set of 500 docs;
However this approach modify the test set everytime: is this a correct approach?
EDIT:
I found many papers that suggest the use of stratified splitting, so that every training and test split has the same proportions of categories in the dataset. I also found that repeated cross validation can generate more accurate measures.
So the approach that I propose is to repeat 10 times every cycle of the previous approach (10x10), using StratifiedShuffleSplit instead of ShuffleSplit.
Do you think i'm doing it well?