Reporting result of classifiers [duplicate]

Question

I am using a binary classifier in my research. I wanna report the obtained result of classifier, but I am not sure about it.

The learning prosedure is as follows:

for (iteration in 1:10) # for more accuracy
{
 1- Divide data to train and test partitions,

 2- Train_result[iteration]: Train classifier using 'train' data using cross-validation and save 'classification error',

 3- Test_result[iteration]: Test the trained classifier in step (1) performance using 'test' data and save 'classification error',

 4- TrainTest_result[iteration]: Use the trained classifier in step (1) to classify all, 'train and test', data and compute its 'classification error'.
}

TrainError_avg = mean(Train_result)
TestError_avg = mean(Test_result)
TrainTest_result_avg = mean(TrainTest_result)

I should report which of the above three measures, 'TrainError_avg', 'TestError_avg', or 'TrainTestError_avg', in my paper?

If I want to plot ROC curve, I should plot ROC curve for which of them? Besides, Should I average between ROC over all iterations?

I appreciate if you could clear these stuff to me.

Answer 1

To answer your first answer let me explain you a bit about the meaning of these statistics that you are computing.

The Training Error is a measure to see how well your model is able to fit to the data training data. The value by itself is not of interest in order to determine how well your model is able to predict data, however it can provide you help in determining the characteristics of your model.

• For example from the training error you can deduce about as to how well your model is able to model data (i.e. does your model have a high variance or bias)
• The training error usually serves as a lower bound to the test error, so by looking at the training error by itself you often have a best-case scenario for the test error.
• From the training error you can deduce how quickly your model is able to fit to the data (i.e. how quick it converges)

The Test Error is a measure to determine how well your model is able to predict unseen data, and thus how well your model is able to generalise when encountering unseen data.

• The test error is particularly interesting in domains as supervised machine learning, in which you already have the classes for your training data, and you really want to use your model to 'predict' classes for unseen data.

Then at last the combination of test and training error is not relevant in most cases, as it is unclear from this statistics which contribution is made from test error and which one from training error. Especially since often test and training datasets differ largely in the amount of instances.

Some other things I personally would recommend looking at is the difference in test and training error for each iteration. This gives you an idea as to how your model performs in different scenarios and provided that you do sufficient amount of iterations you can establish a good indication of the variance of the model.

Then onto your second question. If I am not mistaken the ROC curve is usually not computed from cross-validation results (as you have) but by using a hold-out method. You could simply use 33% percent data for testing and 67% for training from all the data over all iterations.

The ROC that you then obtain over the test data will allow you fine-tune your model by selecting the right threshold to differentiate between the two classes.

At last, on a more general note, cross-validation is usually used only to gain insight in your model performance, its generalisation error, etc. However after having obtained these values and you are sure that you want to put your model in practice you usually re-train it on all the data (training and test) and use a threshold obtain from the hold-out ROC experiment. Don't start fine-tuning the threshold on the training data as you will then greatly overfit.

Hope this helps.