4
$\begingroup$

Let's say I run a lot of univariate OLS regression models, say 200,000, with 50 data points, then cherry pick the best one (highest r-square). If my $p$-value for this model is way less than 1/200,000, do my results still have any explanatory power?

The probability of getting this result when there is no relationship between the variables was extremely low, even running 200,000 instances of it, correct?

I'm generally interested in rules on how/when cherry-picking can still make sense.

$\endgroup$

marked as duplicate by kjetil b halvorsen, Michael Chernick, Peter Flom - Reinstate Monica regression Apr 24 '18 at 23:23

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    $\begingroup$ This is called "multiple comparisons problem" and what you suggest is more or less what is known as "Bonferroni correction". It is a valid procedure, yes. $\endgroup$ – amoeba says Reinstate Monica Aug 7 '15 at 13:07
  • 2
    $\begingroup$ If you are running many univariate regressions, each choosing one from among multiple independent variables with the same dependent variable, then this approach can miss important combinations of independent variables and lose explanatory power. If that's what you're doing, edit the question or pose another that explains what you are trying to accomplish with your study. There are much better ways to proceed in that case. $\endgroup$ – EdM Aug 7 '15 at 13:35
  • $\begingroup$ Do you work in a fixed regressor setting? (I.e., is your regressor matrix $X$ stochastic or deterministic? The latter would be the case for lab experiments.) Depending on the answer, I would recommend one of two methods $\endgroup$ – Jeremias K Feb 17 '16 at 19:33