1
$\begingroup$

I have a big dataset with many predictors. I would like to know the quality of these predictors using cross-validation. However, what I am finding is more so methods that test the reliability of the model as a whole.

Is it possible to check the reliability of specific predictors?

$\endgroup$
3
  • $\begingroup$ I suggest elastic net instead. $\endgroup$
    – Germania
    Aug 5 at 20:27
  • 2
    $\begingroup$ What exactly do you mean by "reliability" in this context? Generally, CV is used to assess predictive value. (Many researchers use "reliability" as a synonym for significant.) You could fit an otherwise identical model w/ & w/o a variable & assess the change in the root mean squared error of the prediction (RMSEP). $\endgroup$ Aug 5 at 20:27
  • $\begingroup$ @gung-ReinstateMonica Thanks for the reply. I am indeed using it as a synonym for significant. Basically, my issue is that with so many data points almost every predictor is significant. I would like to separate those that are genuine effects from those that are Type 1 errors. Is the method you suggest the best way to do this? $\endgroup$
    – Dave
    Aug 5 at 20:35

1 Answer 1

1
$\begingroup$

Generally, cross-validation is used to assess predictive value. From that perspective, you could fit an otherwise identical model with and without a variable and assess the change in the root mean squared error of the prediction (RMSEP).

Many researchers use "reliability" as a synonym for significant. I gather that is your intention here. Without intending any disrespect, I think this is a poor usage. If you haven't done any data snooping / p-hacking, there is no reason to be concerned that many variables are significant at $\alpha = 0.05$ when you have many data. If you want, you could always use a more stringent alpha to reduce the risk of type I errors when you have many data.

$\endgroup$
2
  • $\begingroup$ Thanks very much! This worry over significance is based on a comment from a colleague. Is there a good resource I could cite to say that this isn't an issue? $\endgroup$
    – Dave
    Aug 5 at 20:55
  • 1
    $\begingroup$ @Dave, just an introductory stats book. That $p<\alpha$ provides $\alpha$-level protection against type-I errors is basically the definition. If you want to get fancy, maybe Casella & Berger. $\endgroup$ Aug 6 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.