I validate usage of a clinical cardiovascular score to predict the risk of dementia using data from a longitudinal study. Therefore, my outcome is binary (dementia yes or not) and the independent variable (the score) is continuous, of course I have a whole set of covariates.

I did Cox analysis to assess an association between baseline values and the outcome over time but now I would like to validate the use of the score. I thought about taking a random sub-sample of my cohort to split in training and test and run some sort of validation statistics (i.e. ROC curves) but I have some concerns about this for a number of reasons:

  • My sample is relatively small ($n=2500$), and I am afraid that taking a sub-sample would reduce the power too much.
  • Not sure whether the ROC (or alternatively the somerset) are the best tests in this case, as other tests (like those used in screenings evaluation) may suit better.

How shall I evaluate the use of this score? Can you suggest tests that suit better for the problem?

For data analysis I use Stata.


So you basically want to do a Cross Validation of your dataset.

One type of CV is the Holdout method which divides the data into two parts: Dtest and Dtrain. The model will predict values from Dtest (x → y) and since we know the actual values, that is what x-value correspond to what y-value we can compare the predicted and the actual values and estimate the performance. The subsample is quite arbitrarily Dtrain/Dtest: 70/30.

But your concern about splitting up your dataset into subset is valid. Why? Because when the data is split up into subparts with a dataset that is quite small, in your case N=2500, there's a much higher chance that Dtest and Dtrain is different from one another.

We can solve this by using a k-folded CV. K-folded cross validation divides the dataset into several parts. One of the subparts will be used as test data and the rest (k-1 parts) will be used as training data. The model is iterated (sort of) through each subpart and an error rate will be obtained for each iteration. The mean error rate will be used as a performance value. This solves the problem of having subparts of the data that is not representative of the whole dataset.

The problem with using k-folded cross validation is if the subparts are divided so as to give a model that overfits the data. That is, it predicts the value of the datapoints that are given quite well but when the model is given new data the error rate will be high. This usually happens when there's little data to begin with. To prevent this we can use Repeated cross validation.

Repeated cross validation:

  1. Do CV (which will give mean error rate (Ê)
  2. Reorder the data so as to give different subparts
  3. Repeat 1-2

enter image description here

Figure 1. General idea behind CV.

  • $\begingroup$ I would recommend against repeated cross-validation. See: Vanwinckelen, G., & Blockeel, H. (2011). On estimating model accuracy with repeated cross-validation. Proceedings of BeneLearn and PMLS 2012, 39–44. Retrieved from lirias.kuleuven.be/bitstream/123456789/346385/3/… $\endgroup$ – Kodiologist Oct 20 '15 at 17:08
  • $\begingroup$ This raises some concern. I recently did a course (2015) in machine learning and design of experiments at Uppsala University which taught this as a methodology for validating models, specifically regression models. How recent is this article, are there several of them and is it about to get 'mainstream'? $\endgroup$ – Lennart Oct 20 '15 at 17:19
  • $\begingroup$ It's from 2011. How mainstream those authors' opinion is is hard for me to say; I don't really have my finger on the pulse of the machine-learning community. But my perception is that repeated cross-validation is something people do because it seems intuitive rather because of empirical evidence or strong theory. In that case, Vanwinckelen and Blockeel are among the few who have thought carefully about this issue, so we should listen to them. $\endgroup$ – Kodiologist Oct 20 '15 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.