8
$\begingroup$

I am trying to better understand statistical significance, effect sizes and the like.

I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in large samples. By irrelevant I mean that there is no subject-matter explanation why the regressor should be related to the dependent variable. Thus irrelevance in this post is a pure subject-matter concept and not a statistical one.

I know that a regressor will be statistically significant given a sufficiently large sample unless the population effect is exactly zero (as discussed here). Hence, an irrelevant regressor that appears statistically significant in a large sample has a non-zero effect size in population.

Questions:

  1. How come an irrelevant regressor turns out statistically significant?
  2. Should I look for subject-matter explanation (i.e. try to deny irrelevance) or is this a statistical phenomenon?

This is a continuation of a post where I was trying to clarify how to cure this effect. Meanwhile, here I am asking why it happens in the first place.

|cite|improve this question
$\endgroup$
  • $\begingroup$ This still appears to duplicate your previous post, which itself had already been answered. One answer to the duplicate of that post states " p-values are arbitrary, in that you can make them as small as you wish by gathering enough data." Doesn't that address both #1 and #2? $\endgroup$ – whuber♦ Jan 19 '15 at 22:38
  • 1
    $\begingroup$ Thanks @whuber. I was trying to make it clear that this post addresses a new, different question. I already understand what happens if there truly is an effect in population (the topic of the other post and an older one it duplicates). My questions here are (i) is the reason for the frequent statistical significance of seemingly irrelevant regressors always the same, i.e. that there actually is a population effect; (ii) if not, then what are the alternative reasons; (iii) if yes, then are the effects in population most often due to subject-matter or due to chance. I hope this makes it clearer. $\endgroup$ – Richard Hardy Jan 20 '15 at 7:17
9
$\begingroup$

Questions:

How come an irrelevant regressor turn out statistically significant?

I think it's helpful to think about what happens as your sample size approaches the population itself. Significance testing is meant to give you an idea of whether not an effect exists in the population. This is the reason why when working with census data (that surveys the population), significance testing is meaningless (because, what are you trying to generalize to?).

With that in mind, what does "an effect in the population" mean? It simply means any relationship between variables in the population, regardless of how small (be it a 1-point or 1-person difference), even if that relationship is due to chance and randomness in the universe.

Thus, as your sample approaches the size of the population, significance tests become less and less meaningful because any difference will be "statistically significant". What you would be more interested in then is effect size - which is analogous to "practically significant".

Should I look for subject-matter explanation (i.e. try to deny irrelevance) or is this a statistical phenomenon?

It's a phenomenon - you should look at effect sizes.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ You mention something I was looking for intuitively but had not formulated successfully. The effect in population may be any relationship between variables in the population, regardless of how small (be it a 1-point or 1-person difference), even if that relationship is due to chance and randomness in the universe. If these effects are due to chance, I need not look for any subject-matter explanation. Thanks! $\endgroup$ – Richard Hardy Jan 20 '15 at 7:22
  • $\begingroup$ By the way, your argument makes intuitive sense in cases where one is testing e.g. equality of means or whether treatment effect is exactly zero. But what about regressors' significance in a multiple regression? Could you perhaps state your point in a regression context so that I could more readily follow? $\endgroup$ – Richard Hardy Jan 20 '15 at 8:04
  • 1
    $\begingroup$ Hmm, I think the analogy is directly relevant. For example, if you imagine chocolate consumption has no effect on life expectancy, after adjusting for control variables such as amount of exercise etc., but one person in the population of 6 billion people happens to be an outlier, there will be a population "effect" of chocolate consumption on life expectancy, but the size of the effect will be minuscule. Hopefully that example was helpful, but I had trouble thinking how a regression coefficient would be different from any other parameter. $\endgroup$ – QxV Jan 20 '15 at 13:26
  • 3
    $\begingroup$ I am a bit confused as to why significance testing should be meaningless if you sample the whole "population" (if by population you mean the actual population). If I want to test something about people and my sample is all 7 billion people, then surely I can still run a significance test and it might reject or fail to reject the null hypothesis. I don't see why it should be conceptually meaningless. The "generalization" that you mentioned can refer e.g. to the future generations or something. (CC @Richard.) $\endgroup$ – amoeba says Reinstate Monica Jan 22 '15 at 9:52
  • 1
    $\begingroup$ @amoeba: I think statistical significance testing only makes sense for a sample, not a population. Once we observe the whole population, all our knowledge is perfect: there is no parameter estimation uncertainty or the like. We can measure any relationship exactly. (That does not mean we will have a perfect subject-matter explanation for all relationships, but that's besides the point.) Meanwhile, if you generalize to future generations, that implicitly makes the current population just a sample from the {current+future} population. Then statistical significance testing is back in the game. $\endgroup$ – Richard Hardy Jan 22 '15 at 10:39
2
$\begingroup$

Even if your sample size doesn't approach your population, tiny effects become significant in large samples. This is a consequence of what statistical significance means:

If, in the population from which this sample was taken, the null hypothesis was true, is it (XX%) likely that we would get a test statistic at least this large in a sample of the size we have?

If your question is something about all people on Earth, then if you take a sample of 1,000,000 (not close to 7,000,000,000) even very tiny effects will be significant, because it's very unlikely to find such test statistics in large samples when the null is true.

There are lots of problems with significance testing, discussed in many places. This is one of them. The "cure" is to look at effect sizes and confidence intervals.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks! I think I understand the mechanism how those tiny effects turn into statistically significant ones given a large enough sample. The core question is, why are those tiny effect present in the population to begin with? Do they come about mostly due to "randomness in the universe"? Or do they represent some actual subject-matter relations (not due to chance) that we tend to neglect when we think about them from the subject-matter point of view? $\endgroup$ – Richard Hardy Jan 20 '15 at 12:14
  • 4
    $\begingroup$ I would say the latter. $\endgroup$ – Ben Bolker Jan 20 '15 at 12:59
  • $\begingroup$ @BenBolker Could you please provide some reasoning? That could be very helpful. $\endgroup$ – Richard Hardy Jan 20 '15 at 13:05
  • 3
    $\begingroup$ I agree with Ben. Almost any two variables are going to be related to some degree; and the ones that we stick into models are much more likely to be related. We don't (and shouldn't) just toss junk into models. $\endgroup$ – Peter Flom - Reinstate Monica♦ Jan 20 '15 at 19:09
2
$\begingroup$

I have borrowed some insight from @QxV to provide an explanation of presence of a population effect even if subject-matter knowledge suggests no such effect.

Suppose there is a population-generating process (PGP) that generates populations with features $y$ and $x$. The PGP formula is such that $y$ and $x$ are independent up to a random error term. Precisely due to this random error term, any finite realization $y_{realized}$ and $x_{realized}$ has zero probability of exact uncorrelatedness, i.e. $P(y_{realized} \perp x_{realized})=0$. If so, with probability one there is a population effect. That is how effects come about in population.

Once a population effect exists, it is a matter of sample size when we will detect it in the sample and when it will become statistically significant.

|cite|improve this answer
$\endgroup$
2
$\begingroup$

In addition to the excellent answers already posted, I will try from another point of view. All models are approximations, in some sense ... Look at some regression model, and some irrelevant variable is significant. What can explain it?

  1. Maybe it just is not irrelevant, that todays scientific consensus on that matter is just wrong. Apart from that:

  2. It could be a stand-in or proxy for some omitted variable which is relevant, and which is correlated with the irrelevant variable.

  3. Some relevant variable, included linearly in the model, could be acting non-linearly, and your irrelevant variable could be a stand-in for that part of the relevant variable.

  4. Some interaction between two relevant variables is important, but not included in the model. Your irrelevant variable could be a stand-in for that omitted interaction.

  5. The irrelevant variable could just be very highly correlated with some important variable, leading to negatively correlated coefficients. This could be important especially if there are measurement errors in this variables.

  6. There could be some observations with very high leverage, leading to strange estimates.

Surely others ... an important point is that a linear regression model could be a very good approximation with a small sample, only large effects will be significant. But a larger sample will lead to lower variance, but it cannot reduce bias due to approximations. So with larger samples those inadequacies of the model becomes manifest, and will eventually dominate over variance.

|cite|improve this answer
$\endgroup$
-1
$\begingroup$

No. Irrelevant regressors do not become statistically significant as sample size increases. Try the following code in R.

y <- rnorm(10000000)

x <- rnorm(10000000)

summary(lm(y~x))

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Did you repeat that code, say, 1000 times (or more) and see what will happen? $\endgroup$ – kjetil b halvorsen Mar 12 at 12:53
  • $\begingroup$ You will have a 5% false positive rate, as expected. But the same is obtained regardless of sample size $\endgroup$ – David Mar 12 at 13:54
  • $\begingroup$ See my answer for another viewpoint $\endgroup$ – kjetil b halvorsen Mar 12 at 15:40
  • $\begingroup$ I understand your post, and for "kind-of-linear" relationships it is true (as it is for "actually-linear" relatinoships) However, in you change 10000000 for 10 in my code, you are not now less likely to obtain a false positive in the F-test $\endgroup$ – David Mar 13 at 7:44
  • $\begingroup$ Thank you for your answer! While correct on its own, I think it misses the point of the question. The question is motivated by the observation that we often find statistically significant relationships that have no subject-matter explanation. $\endgroup$ – Richard Hardy Mar 23 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.