I am a statistics novice. I therefore kindly ask for your understanding with this basic question:

I want to cross validate the results of my logistic regression. So far I have divided the overall sample into two sub-samples (testing and holdout). I have run the logistic regression on the testing sample and afterwards run the same model on the holdout sample. I now have two regression outputs, but I am not sure on what to do next. How does the cross-validation work? Can someone please explain me how I should proceed. Which measures do I need to compare and when can I say that the model is valid?

Thank you very much in advance

  • 1
    $\begingroup$ You can never say that your model is valid, actually it is the opposite "all models are wrong" quoting G.E.P. Box. $\endgroup$ – Tim Oct 10 '16 at 13:48

What you've described so far is the start of one cross-validation step. Here's the generic procedure:

1) Divide data set at random into training and test sets.

2) Fit model on training set.

3) Test model on test set.

4) Compute and save fit statistic using test data (step 3).

5) Repeat 1 - 4 several times, then average results of all step 4.

Cross-validation is one method of trying to reduce overfitting (optimism) in a fitted model. Typically these are regression-based models used for prediction. By randomly dividing the data set as above, there is less certainly about the final model, but on aggregate, the process tells you something about how the model might generalize to a new independent data set. This is one way of performing model validation.

There are several types of cross-validation that can be broadly divided into "exhaustive" or "non-exhaustive" methods. Exhaustive methods mean that every possible training/test split is performed. These are the leave-one-out or leave-n-out methods in which one data point (or n points) are used for the test set. The non-exhaustive methods divide the data into equally sized groups (k-fold cross-validation) or use more complex methods to sample the data.

The measure you should use for binary classification depend on what you are trying to do, and there are lots of measures of fit that might be of interest (mean absolute deviation, Brier score, c-statistic, $R^2$, positive/negative predictive value, precision, recall). You'll need to look at related literature or tell is more information to decide which of these you will need.

The cross-validation procedure only tells you about the degree of overfitting (optimism) in the original model. Once it's done, the model validation has been characterized. It's up to you to determine whether the amount of overfitting (optimism) is acceptable or not using existing literature or expert judgement. Model validity in this sense only means the model makes sense and has an acceptable level of error, not that it is the perfect or best model (which is most likely impossible to ever know).

  • $\begingroup$ If I understand it correctly the fit statistics of the training set are not really relevant... what matters are the fit statistics of the test set. In case I only do a simple validation (without step 5) I would not even have to average the result. Is this correct? $\endgroup$ – TheRedhead Oct 10 '16 at 14:12
  • $\begingroup$ The first part of your comment if correct. If you instead computed statistics on the training set, then that's a related procedure called a jackknife. You absolutely do need to repeat the cross-validation process multiple times to get reliable estimates. $\endgroup$ – prince_of_pears Oct 10 '16 at 14:26
  • $\begingroup$ Thanks for the clear answers. One last detailed question: In step 5), do I need to average all the fit statistics including the regression coefficients? If I choose a stepwise selection approach for my factors it may happen that one of the n-models has other variables in it... or do I have to make sure that the same model is run several times? $\endgroup$ – TheRedhead Oct 10 '16 at 14:40
  • $\begingroup$ While you can use CV for variable selection, you should probably avoid those and just focus on the fit statistic. Frank Harrell has a related answer here stating the same thing and also has a great discussion elsewhere of why stepwise variable selection is a bad idea. stats.stackexchange.com/questions/22085/… $\endgroup$ – prince_of_pears Oct 10 '16 at 14:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.