# Is it possible to use cross-validation to estimate the reliability of a specific predictor?

I have a big dataset with many predictors. I would like to know the quality of these predictors using cross-validation. However, what I am finding is more so methods that test the reliability of the model as a whole.

Is it possible to check the reliability of specific predictors?

• I suggest elastic net instead. Aug 5 at 20:27
• What exactly do you mean by "reliability" in this context? Generally, CV is used to assess predictive value. (Many researchers use "reliability" as a synonym for significant.) You could fit an otherwise identical model w/ & w/o a variable & assess the change in the root mean squared error of the prediction (RMSEP). Aug 5 at 20:27
• @gung-ReinstateMonica Thanks for the reply. I am indeed using it as a synonym for significant. Basically, my issue is that with so many data points almost every predictor is significant. I would like to separate those that are genuine effects from those that are Type 1 errors. Is the method you suggest the best way to do this?
– Dave
Aug 5 at 20:35

Many researchers use "reliability" as a synonym for significant. I gather that is your intention here. Without intending any disrespect, I think this is a poor usage. If you haven't done any data snooping / p-hacking, there is no reason to be concerned that many variables are significant at $$\alpha = 0.05$$ when you have many data. If you want, you could always use a more stringent alpha to reduce the risk of type I errors when you have many data.
• @Dave, just an introductory stats book. That $p<\alpha$ provides $\alpha$-level protection against type-I errors is basically the definition. If you want to get fancy, maybe Casella & Berger. Aug 6 at 0:55