1
$\begingroup$

I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared deviation).

As the first step, I can randomly split my data set into two equal parts. I use the first part to train each of the two models and then I evaluate the two trained models using the second data set. In that way I get two out-of-sample errors. Then I could say that the model corresponding to the smaller error is better. However, how can I be sure that the results are statistically significant? For example, it can be the case that the first model wins for one split of the data and looses for another split.

I thought that, as a workaround, I could take all the deviations between the targets and the corresponding predictions and then resample from these deviations. Then I check how frequently one model is better / worse than another one. However, this way I do not count for the randomness determined by the limited size of the training set. In other words, it might be the case that model 1 is statistically significantly better than the model 2 if both models are trained on the first 100 randomly selected data points but if we use other 100 data points, model 2 can be better than model 1.

I guess that the main cause of my problem is that the measure of accuracy is random because of two reasons: (1) the limited sizes of the evaluation set and (2) limited size of the training set. So, I need to have a way to treat this randomness properly.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Read up on (leave-one-out) cross-validation and model selection criteria. Note: Leave-one-out cross-validation is asymptotically equivalent to the AIC model selection criterion. $\endgroup$
    – bbrot
    Dec 4, 2018 at 18:18
  • $\begingroup$ @bbrot, I know about leave-one-out cross-validation. Unfortunately, I do not see how it solves my problem. For each data point I do get an out-of-sample prediction and each model gets an average error this way. But still, I only have "sample error", which is a random number and one of them can be smaller than another one just by chance (and not because the corresponding population error is smaller). Moreover, I do not know what results would I get if I have another data set of the same size (which I do not have). $\endgroup$
    – Roman
    Dec 5, 2018 at 7:53
  • $\begingroup$ @bbrot: Roman is right - LOOCV is one of the few cross validation procedures that never allows to distinguish limited-number-of-test-cases-variance from limited-number-of-training-cases-variance (instability). $\endgroup$
    – cbeleites unhappy with SX
    Dec 5, 2018 at 14:26

1 Answer 1

Reset to default
2
$\begingroup$

You are right that your validation/verification results are subject to (at least) these two sources of variance:

the measure of accuracy is random because of two reasons: (1) the limited sizes of the evaluation set and (2) limited size of the training set.

Variance (2) is commonly called model instability.

Practically speaking, there are 3 situations:

  1. Var (1) >> Var (2): you can use Var (1) as approximation for the total variance.

  2. Var (1) << Var (2): you can use Var (2) as approximation for the total variance.
    However, you may want to reconsider the meaning of this finding: the predictions of the model in question is unstable wrt. exchanging some training cases for other training cases from the same population. Maybe you should improve your training first in order to arrive at a stable models.

  3. variance (1) and (2) are in the same order of magnitude. Both of them need to be taken into account for the decision.
    Again, think whether the observed level of instability is acceptable for your application.

You can measure Var (2) rather directly with iterated/repeated cross validation (see e.g. our paper: Beleites, C. & Salzer, R. Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 2008, 390, 1261-1271), but also out-of-bootstrap allows you to separate the two variance sources.

When formulating the test, keep in mind that Var (2) is likely or even expected to vary between models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.