I am running a multiple regression model and looking to use AIC and BIC to select models. However I notice that both measures do not consider the number of variables we can choose from but only consider the number of variables chosen. If I have many many variables to choose from, chances are I will find something highly correlated with what I am trying to model, just by luck. Is there a measure that considers how many variables we can choose from?
I think simple cross validation is the best fit.
Both AIC and BIC consider the balance between model complexity and the amount of information available. With more data, more complex models can be learned. However, this balance is fixed and not based on the data.
Cross validation is based on the data. It also balances model complexity with the amount of information available. With more data more complex models can be learned. The performance on unseen data quantifies how well the model works. Implicitly, models that are to complex (overfitting) are penalized because they make bad predictions.
In the case of many variables the highly correlated ones can be chosen during training. During testing however it becomes apparent that the learned relations do not generalize to unseen data.
Another advantage of cross validation is that you can choose your own performance measurement.