There are a lot of criterion bases or test based tools to compare different linear models and perform variable selection (for example, we have adjusted R squared, AIC, F-test, and so). However, as far as I know, those tools can't be used if the number of cases in both models is different.
In my data I have a lot of missing values. Therefore, adding a predictor usually means losing some cases - and adding a lot of predictors means losing a lot of cases.
Then, my question is how can I choose between (let's say) a model with 50,000 cases and 500 predictors and another one with 25,000 cases and 1000 predictors - and all those in between:
Should I resort to cross validation?
Or should I do an F-test with the smaller dataset and both models using the standard errors from each model in its original dataset?
Is there a criterion based procedure suitable for linear models with different numbers of cases?
Edit: For the scope of this question, values can be assumed to be missed completely at random.