17
$\begingroup$

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum the confusion matrices?

$\endgroup$
9
$\begingroup$

If you are testing the performance of a model (i.e. not optimizing parameters), generally you will sum the confusion matrices. Think of it like this, you have split you data in to 10 different folds or 'test' sets. You train your model on 9/10 of the folds and test the first fold and get a confusion matrix. This confusion matrix represents the classification of 1/10 of the data. You repeat the analysis again with the next 'test' set and get another confusion matrix representing another 1/10 of the data. Adding this new confusion matrix to the first now represents 20% of your data. You continue until you have run all your folds, sum all your confusion matrices and the final confusion matrix represents that model's performance for all of the data. You could average the confusion matrices but that doesn't really provide any additional information from the cumulative matrix and may be biased if your folds are not all the same size.

Note -- this assumes non-repeated sampling of your data. I'm not completely certain if this would be different for repeated sampling. Will update if I learn something or someone recommends a method.

$\endgroup$
  • $\begingroup$ Thank you, cdeterman. What about model selection (i.e.optimizing tuning parameters)? $\endgroup$ – John M Mar 20 '16 at 21:48
  • $\begingroup$ @JohnM then you are looking at each fold independently to get an indication of what the best parameters for the full model should be. You may want to look in to nested cv if you want to combine both. $\endgroup$ – cdeterman Mar 21 '16 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.