0
$\begingroup$

I'm looking for confirmation on my understanding of a 'statistically significant' effect.

For example, let's say that eating apples has a statistically significant positive effect on health (given that you can quantify health in some way). This does not mean that everyone will benefit from eating apples. Rather, it means the average individual from the population will benefit from eating apples.

This would imply that that the closer a single individual is to the the mean, the more likely they are to benefit from eating apples. The farther away an individual is from the mean, the less probable the tangible benefit from eating apples becomes.

Finally, if the above is true, how can statistically significant effects be applied at the individual level when individual effects require comparison to the mean? For example, the statistically significant result for apples is only relevant insofar that you are near the mean, but we have no idea how far any individual is from the mean without some additional metric. Is there a metric for this?

I suspect this line of thought is related to Stephen Jay Gould's "The Median Isn't the Message".

Improve this question
$\endgroup$
8
  • $\begingroup$ The conclusion supported by the significant effect is that the mean effect is not 0. This doesn't imply what portion of the distribution will be benefited. It could be it benefits only those in the lowest 1% of the distribution. It doesn't imply those close to the mean will be affected. $\endgroup$
    – David Lane
    Jan 11, 2017 at 18:07
  • $\begingroup$ The lowest 1% of the population distribution, not the distribution of sample means right? If so, how are p-values useful if they ignore the population distribution? I understand how the CLT guarantees a normal distribution of the sample means but that doesn't seem very useful if we ultimately care about the population distribution. $\endgroup$
    – Alex Petralia
    Jan 11, 2017 at 19:31
  • $\begingroup$ Also if the population distribution is normal, is it possible to have a significant effect if only the bottom 1% of the distribution is affected? $\endgroup$
    – Alex Petralia
    Jan 11, 2017 at 19:37
  • 1
    $\begingroup$ @David Not all tests that assume normality are robust to violations of it. You can certainly make an argument that t-tests are somewhat level-robust, but that's hardly the gamut of normal-theory tests (consider F-tests for equality of variance for example), and level robustness alone may not be sufficient -- it depends on the circumstances: the power properties (e.g. as indicated by relative efficiency against common alternative choices) may become poor in large samples when looking for small effects, which may or may not be a big issue. $\endgroup$
    – Glen_b
    Jan 12, 2017 at 1:18
  • 1
    $\begingroup$ @glen_b The quantile of what exactly? What distribution are we talking about? For example, the top 10 percent of those affected by apple eating? Or the top 10 percent of some other variable in the population (like health, wealth, etc.)? If the former, we may only see the effect in the top 10 percent of the treatment effect? $\endgroup$
    – Alex Petralia
    Jan 20, 2017 at 3:11

1 Answer 1

Reset to default
0
$\begingroup$

You seem to have answered your own question by that linked paper. A quote:

If a little learning could ever be a dangerous thing, I had encountered a classic example. Attitude clearly matters in fighting cancer. We don’t know why (from my old-style materialistic perspective, I suspect that mental states feed back upon the immune system). But match people with the same cancer for age, class, health, and socio­economic status, and, in general, those with positive attitudes, with a strong will and purpose for living, with commitment to struggle, and with an active response to aiding their own treatment and not just a passive acceptance of anything doctors say tend to live longer. A few months later I asked Sir Peter Medawar, my personal scientific guru and a Nobelist in immunology, what the best prescription for success against cancer might be. “A sanguine personality,” he replied. Fortunately (since one can’t reconstruct oneself at short notice and for a definite purpose), I am, if anything, even-tempered and confident in just this manner.

A more technical answer could include terms like unit-treatment interaction, in your example, interactions between apple eating and some measurable individual characteristic. Maybe somebody could have "apple allergy" (never heard about that). Some links on this site: Unequal variance in randomized experiments to compare treatment with control?, Unequal variance in randomized experiments to compare treatment with control?,

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.