This question already has an answer here:

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a 5% significance level will pretty much never make me entitled to reject the null (of the parameter being equal to zero), i.e. everything is significant.

I should probably use another significance level, which is smaller than 5%, and my question is if there is any literature on picking the right significance level, or how would you guys handle this problem? Should I simply stick with the 5% significance level?



marked as duplicate by gung - Reinstate Monica, Nick Cox, Peter Flom - Reinstate Monica Jan 8 '14 at 10:25

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ You parameter is never exactly zero; why do you need to reject the obviously wrong? I'll give you a reference to Tukey on how to formulate good hypotheses later today. How do you estimate your model "using linear regression or MLE"? The rule of 5% is by no means a standard one - just a convenient one - can refer you to a good paper co-authored by Peter Hall but again later. $\endgroup$ – Hibernating Jan 8 '14 at 4:14
  • $\begingroup$ Thanks for the good answer. I'm not interested in the parameter being exactly equal to zero, this will happen with probability 0, but I'm interested whether it is indistinguishable from zero. I'm referring to the 5 % significance level as a standard rule, because in 99 % of the articles in economics/econometrics this is used without further discussion. Regarding my question, it is not of interest to me whether I do MLE or linear regression as this is just a question referring to standard inference, i.e. doing t-test, chi-square test or similar. $\endgroup$ – user37008 Jan 8 '14 at 12:51
  • $\begingroup$ I am afraid you need to ask another question. A quick search shows something like How to formulate a good null hypothesis has not been asked yet. This will allow me to deploy Tukey. Ask about history of that golden rule of stats (p-value > 0.05) and I will refer you to Hall. In many situations hypothesis testing is less than appropriate regardless of the size of your data. $\endgroup$ – Hibernating Jan 9 '14 at 0:57
  1. The first (default) method to correct for the "multiple comparisons" problem is to use a Bonferroni adjustment to the p-value. The new $\alpha$ level is $\alpha^*=\frac{\alpha}{\#tests}$. Thus, divide the Type I error level of $\alpha=0.05$ by the number of tests you have.

  2. Use the Benjamini-Hochberg False Discovery Rate approach (1995), which is less conservative.

  • 1
    $\begingroup$ I assumed this question was about the increased likelihood that even a negligibly small or spurious effect could achieve significance in a large data problem, but you might be right to address the separate issue of multiple comparisons, depending on whether the OP wants to control for this kind of error inflation. See also an issue with multiple comparisons corrections. $\endgroup$ – Nick Stauner Jan 8 '14 at 4:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.