6
$\begingroup$

I have a project for which I had to do a 5-fold cross-validation. The dataset comprises of 5 days, so we take one day as our training set and apply on the remaining 4, rinse and repeat till we get full coverage. Pretty standard stuff.

Now, here's my question, once I finished the cross-validation, I am left with True Positive and False Positive rate for each one my runs. Is there a standard way to show this data visually that would summarize the results ? In particular I want to show that the technique is fairly stable and about a day's worth of data is enough to train it successfully.

$\endgroup$
11
$\begingroup$

box and whisker plots are commonly used to visually compare and summarize cross validation results.

enter image description here

Here is an example, taken from the cvTools package in R.

library(cvTools)
## set up folds for cross-validation
folds <- cvFolds(nrow(coleman), K = 5, R = 50)
## compare LS, MM and LTS regression
# perform cross-validation for an LS regression model
fitLm <- lm(Y ~ ., data = coleman)
cvFitLm <- cvLm(fitLm, cost = rtmspe,
folds = folds, trim = 0.1)

...

# plot results for the MM regression model
bwplot(cvFitLmrob)
# plot combined results
bwplot(cvFits)
$\endgroup$
  • $\begingroup$ could you explain what the X and Y axis are ? $\endgroup$ – creatiwit Nov 2 '12 at 19:17
  • 1
    $\begingroup$ In this case, x represents the rms prediction errors over 5 repeated out of sample folds, across 3 different models. The corresponding model labels are represented on the y axis. $\endgroup$ – pat Nov 2 '12 at 19:51
  • $\begingroup$ silly follow-on question, but do the parameters change for a logistic regression? e.g. would the code for logit just be: fitLm <- glm(Y ~ ., data = coleman, family = 'binomial'); cvFitLm <- cvLm(fitLm, cost = rtmspe,folds = folds, trim = 0.1) $\endgroup$ – NiuBiBang Jul 23 '14 at 17:11
  • $\begingroup$ This is a great solution, the only change I would make would be that since it appears that the user only has 5 fixed folds of cross-validation rather than the 5x50 repeated folds as specified by the answer, it might be better to plot the errors as points rather than as a boxplot. Boxplots can be misleading for small numbers of data points, so it is often more informative to view independent instances until there get to be so many that the points become indistinguishable. $\endgroup$ – Barker Aug 29 '16 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.