2
$\begingroup$

There are a lot of criterion bases or test based tools to compare different linear models and perform variable selection (for example, we have adjusted R squared, AIC, F-test, and so). However, as far as I know, those tools can't be used if the number of cases in both models is different.

In my data I have a lot of missing values. Therefore, adding a predictor usually means losing some cases - and adding a lot of predictors means losing a lot of cases.

Then, my question is how can I choose between (let's say) a model with 50,000 cases and 500 predictors and another one with 25,000 cases and 1000 predictors - and all those in between:

Should I resort to cross validation?

Or should I do an F-test with the smaller dataset and both models using the standard errors from each model in its original dataset?

Is there a criterion based procedure suitable for linear models with different numbers of cases?


Edit: For the scope of this question, values can be assumed to be missed completely at random.

$\endgroup$
2
  • $\begingroup$ If the missingness mechanism is such that the missing is informative then dropping the cases with missing is likely to lead to bias. I think this is going to make it hard to give a universal best buy. $\endgroup$ – mdewey Dec 5 '17 at 13:58
  • $\begingroup$ I'm aware of it. However, for the scope of this question we can assume that values are missing completely at random. In fact, that's nearly true, since a great deal of my missingness comes from weather stations being established and shut down - and not destroyed by extreme meteorological events. $\endgroup$ – Pere Dec 5 '17 at 14:49
1
$\begingroup$

Since you can assume that values are missing at random, the best solution would probably be to compare both models (using F-tests, likelihood ratio tests, mean absolute error, median absolute error, or other such approaches) using the data subset that has all predictors present in both models.

Here are two other solutions to consider:

  • Create a holdout data subset that has all predictors. Fit the models to the remaining data (which can be different for the two models). Compare the prediction error of the models on the holdout dataset.
  • One that I would not strongly recommend but could work, in principle. Fit the two models to their separate datasets. Compare the models fitted to the different datasets based on their prediction error (e.g. mean absolute error, median absolute error). Even though the datasets are different, differences in prediction error should be meaningful if the values are missing at random.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.