# How to validate sentiment classification and compare different algorithms

I need to compare SVM and NB about sentiment classification by evaluating accuracy, precision and recall measures. I have 1500 manually classified documents, and I would know which is the best way to compare these two algorithms, also increasing training set size from 100 to 1000 documents. I'm using scikit-learn and there are different methods, like KFold, ShuffleSplit, StratifiedKFold, StratifiedShuffleSplit. Which is the right one for my needs?

My approach is to use ShuffleSplit, using a fixed random seed, to first shuffle docs and then select 10 iterations of X training documents and 500 test documents, where X varies from 100 to 1000.

So, I have:

• cycle 1) 10 iterations with training set of 100 docs and test set of 500 docs;
• cycle 2) 10 iterations with training set of 100 docs and test set of 500 docs;
• ...
• cycle 10) 10 iterations with training set of 1000 docs and test set of 500 docs;

However this approach modify the test set everytime: is this a correct approach?

EDIT:

I found many papers that suggest the use of stratified splitting, so that every training and test split has the same proportions of categories in the dataset. I also found that repeated cross validation can generate more accurate measures.

So the approach that I propose is to repeat 10 times every cycle of the previous approach (10x10), using StratifiedShuffleSplit instead of ShuffleSplit.

Do you think i'm doing it well?

The best generic way (more on that later) to compare the effectiveness of different classifiers in your data set is with receiver operating characteristics (ROC) curves and area under the ROC curve (AUC). See this paper. Briefly, the metric depends on the false positive and true positive rates (fpr/tpr) so you don't need to take special care making a 50/50 positive/negative data set and deal with the associated problems.

To make ROC curves in sklearn, first have your classifiers return the predicted probabilities:

classifier1 = svm.SVC(kernel='linear', probability=True)
probas_1 = classifier1.fit(X_train, y_train).predict_proba(X_test)


Repeat for classifier2, classifier3, etc.

Next, use sklearn functions to find the fpr, tpr, and auc:

# Compute ROC curve and area the curve
from sklearn.metrics import roc_curve, auc
fpr1, tpr1, thresholds1 = roc_curve(y_test, probas_1[:, 1])
roc_auc1 = auc(fpr1, tpr1)


Again, repeat this for each classifier you are preparing.

Then plot:

# Plot ROC curve
pl.clf()
pl.plot(fpr1, tpr1, label='SVC (area = %0.2f)' % roc_auc1)
pl.plot(fpr2, tpr2, label='KNN(3) (area = %0.2f)' % roc_auc2)
pl.plot(fpr3, tpr3, label='KNN(21) (area = %0.2f)' % roc_auc3)
pl.plot([0, 1], [0, 1], 'k--')
pl.xlim([0.0, 1.0])
pl.ylim([0.0, 1.0])
pl.xlabel('False Positive Rate')
pl.ylabel('True Positive Rate') 