I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is there anything else needed to be included in my report?
$\begingroup$
$\endgroup$
8
-
5$\begingroup$ It depends a lot on what you mean by "got statistically significant results consistent with hypothesis". If your hypothesis is that wind is produced by trees and your experiment shows that in 100% of observations when trees were moving their branches, there was wind, you find it statistically significant and voila your conclusion is proven. Which is obviously wrong. So, this might be one of those cases. $\endgroup$– sashkelloCommented Dec 11, 2013 at 5:59
-
1$\begingroup$ You really would need a follow up study to safely declare a "significant finding" with credence - using a well designed data collection, same model, and same hypothesis test. Also you need to ensure that your current data set represents the "general population" you are making a claim about with a significant finding (this is a key problem for inference with "big data") $\endgroup$– probabilityislogicCommented Dec 11, 2013 at 8:30
-
1$\begingroup$ Surely the answer is as simple as 'correlation is not causation'? $\endgroup$– FractionalCommented Dec 11, 2013 at 15:00
-
1$\begingroup$ Here's my favorite one: People who eat more rice beget more children. Checking the whole world population, you will get statistically signifcant results... $\endgroup$– Karoly HorvathCommented Dec 11, 2013 at 22:19
-
4$\begingroup$ Great answers, but I am surprised no one suggested the obvious solution: Ask him/her. Whenever someone tells you you are wrong about your work or something else you care about, just ask. Telling someone he is wrong because X, y and Z is cool - it's a learning opportunity. But just telling someone he is wrong and dashing off is a dick move. $\endgroup$– SylverCommented Dec 12, 2013 at 7:38
|
Show 3 more comments
7 Answers
$\begingroup$
$\endgroup$
6
Hypothesis testing versus parameter estimation
Typically, hypotheses are framed in a binary way. I'll put directional hypotheses to one side, as they don't change the issue much. It is common, at least in psychology, to talk about hypotheses such as: the difference between group means is or is not zero; the correlation is or is not zero; the regression coefficient is or is not zero; the r-square is or is not zero. In all these cases, there is a null hypothesis of no effect, and an alternative hypothesis of an effect.
This binary thinking is generally not what we are most interested in. Once you think about your research question, you will almost always find that you are actually interested in estimating parameters. You are interested in the actual difference between group means, or the size of the correlation, or the size of the regression coefficient, or the amount of variance explained.
Of course, when we get a sample of data, the sample estimate of a parameter is not the same as the population parameter. So we need a way of quantifying our uncertainty about what the value of the parameter might be. From a frequentist perspective, confidence intervals provide a means of doing, although Bayesian purists might argue that they don't strictly permit the inference you might want to make. From a Bayesian perspective, credible intervals on posterior densities provide a more direct means of quantifying your uncertainty about the value of a population parameter.
Parameters / effect sizes
Moving away from the binary hypothesis testing approach forces you to think in a continuous way. For example, what size difference in group means would be theoretically interesting? How would you map difference between group means onto subjective language or practical implications? Standardised measures of effect along with contextual norms are one way of building a language for quantifying what different parameter values mean. Such measures are often labelled "effect sizes" (e.g., Cohen's d, r, $R^2$, etc.). However, it is perfectly reasonable, and often preferable, to talk about the importance of an effect using unstandardised measures (e.g., the difference in group means on meaningful unstandardised variables such as income levels, life expectancy, etc.).
There's a huge literature in psychology (and other fields) critiquing a focus on p-values, null hypothesis significance testing, and so on (see this Google Scholar search). This literature often recommends reporting effect sizes with confidence intervals as a resolution (e.g., APA Task force by Wilkinson, 1999).
Steps for moving away from binary hypothesis testing
If you are thinking about adopting this thinking, I think there are progressively more sophisticated approaches you can take:
- Approach 1a. Report the point estimate of your sample effect (e.g., group mean differences) in both raw and standardised terms. When you report your results discuss what such a magnitude would mean for theory and practice.
- Approach 1b. Add to 1a, at least at a very basic level, some sense of the uncertainty around your parameter estimate based on your sample size.
- Approach 2. Also report confidence intervals on effect sizes and incorporate this uncertainty into your thinking about the plausible values of the parameter of interest.
- Approach 3. Report Bayesian credible intervals, and examine the implications of various assumptions on that credible interval, such as choice of prior, the data generating process implied by your model, and so on.
Among many possible references, you'll see Andrew Gelman talk a lot about these issues on his blog and in his research.
References
- Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological methods, 5(2), 241.
- Wilkinson, L. (1999). Statistical methods in psychology journals: guidelines and explanations. American psychologist, 54(8), 594. PDF
answered Dec 11, 2013 at 5:26
-
14$\begingroup$ Further to Jeromy's comment, could I recommend that you have a read Ziliac and McCloskey's essay on the cult of statistical significance. It's not the most mind-blowing statistics, but it does provide thoughtful--and entertaining--discussion of why effect sizes, practical significance, and loss functions are extremely important. deirdremccloskey.com/docs/jsm.pdf $\endgroup$– JimCommented Dec 11, 2013 at 5:33
-
$\begingroup$ I think maybe sometimes p should be set lower than .05. Thank you all: gung, Jeromy and Jim $\endgroup$– Jim VonCommented Dec 11, 2013 at 5:59
-
1$\begingroup$ On Ziliak [NB] and McCloskey: If you are busy, read phil.vt.edu/dmayo/personal_website/… first. If you are not busy, still read it first. $\endgroup$– Nick CoxCommented Dec 11, 2013 at 12:30
-
$\begingroup$ You're welcome, @JimVon. FWIW, I sometimes think p should be set higher than .05. It just depends. $\endgroup$ Commented Dec 11, 2013 at 14:34
-
1$\begingroup$ Glad to see Dr. Gelman get namedropped here. Apparently he doesn't even like reporting p-values, let alone using them for serious inference. He also makes a good case for standardizing all of your variables as a matter of course. $\endgroup$ Commented Dec 18, 2013 at 9:41
$\begingroup$
$\endgroup$
2
Just to add to the existing answers (which are great, by the way). It is important to be aware that statistical significance is a function of sample size.
When you get more and more data, you can find statistically significant differences wherever you look. When the amount of data is huge, even the tiniest effects can lead to statistical significance. This does not imply said effects are meaningful in any practical way.
When testing for differences, $p$-values alone are not enough because the required effect size to produce a statistically significant result decreases with increasing sample size. In practice, the actual question is usually whether there is an effect of a given minimal size (to be relevant). When samples become very large, $p$-values become close to meaningless in answering the actual question.
answered Dec 11, 2013 at 8:32
-
$\begingroup$ This is the point adressed in my slide 13 :) $\endgroup$ Commented Dec 11, 2013 at 13:14
-
6$\begingroup$ +1 for this. People not realizing significance is a function of sample size drives me nuts. $\endgroup$– FomiteCommented Dec 11, 2013 at 18:36
$\begingroup$
$\endgroup$
2
If there was a reasonable basis for suspecting your hypothesis might be true before you ran your study; and you ran a good study (e.g., you didn't induce any confounds); and your results were consistent with your hypothesis and statistically significant; then I think you are fine, as far as that goes.
However, you shouldn't think that significance is all that is important in your results. First, you should look at the effect size as well (see my answer here: Effect size as the hypothesis for significance testing). You might also want to explore your data a bit and see if you can find any potentially interesting surprises that might be worth following up on.
answered Dec 11, 2013 at 4:53
-
$\begingroup$ You mean the hypothesis should be reasonable? And how to judge whether my hypothesis will lead meaningless data analysis? “Potentially interesting surprises” should be revealed by Post-hoc? $\endgroup$– Jim VonCommented Dec 11, 2013 at 5:05
-
$\begingroup$ What I mean is, presumably there was some legitimate reason to run the study in the 1st place. Current theoretical knowledge &/or recent studies suggested your hypothesis might be true. Your hypothesis isn't likely to "lead to meaningless data analysis" unless it is incoherent. Potentially interesting surprises / features of your data could very well be discovered post-hoc; the fact that they are surprises implies you didn't know they would occur when you planned the study. The issue regarding "post-hoc" is whether to believe the surprises--they need to be confirmed by future research. $\endgroup$ Commented Dec 11, 2013 at 5:30
$\begingroup$
$\endgroup$
1
Before reporting this and this and this and this, start by formulating what do you want to learn from you experimental data. The main problem with usual hypothesis tests (these tests we learn at school...) is not the binarity: the main problem is that these are tests for hypotheses which are not hypotheses of interest. See slide 13 here (download the pdf to appreciate the animations). About effect sizes, there's no general definition of this notion. Frankly I would not recommend to use this for non-expert statisticians, these are technical, not natural, measures of "effect". Your hypothesis of interest should be formulated in terms understandable by the laymen.
answered Dec 11, 2013 at 7:37
-
2$\begingroup$ One small addition - the null hypothesis should actually mean something outside the context of the current data analysis for standard HT to apply. It shouldn't be "invented" so that you have something to reject in favour of your theory/finding. $\endgroup$ Commented Dec 11, 2013 at 8:20
$\begingroup$
$\endgroup$
2
I'm far from an expert on statistics, but one thing that has been emphasised in the stats courses I have done to date is the issue of "practical significance". I believe this alludes to what what Jeromy and gung are talking about when referring to "effect size".
We had an example in class of a 12 week diet that had statistically significant weight loss results, but the 95% confidence interval showed a mean weight loss of between 0.2 and 1.2 kg (OK, data was probably made up but it illustrates a point). While "statistically significantly"" different from zero, is a 200gram weight loss over 12 weeks a "practically significant" result to an overweight person trying to get healthy?
-
$\begingroup$ This is the point following my slide 13 :) $\endgroup$ Commented Dec 11, 2013 at 13:15
-
2$\begingroup$ This is also an example of testing the "wrong" null hypothesis. Its not the conclusion you're interested in. A better hypothesis test would be that the weight loss is less than 5kg vs greater than 5kg. $\endgroup$ Commented Dec 11, 2013 at 20:25
$\begingroup$
$\endgroup$
This is impossible to answer accurately without knowing more details of your study and the person's criticism. But here's one possibility: if you've run multiple tests, and you choose to focus on the one that came out at p<0.05 and ignore others, then that "significance" has been diluted by the fact of your selective attention to it. As an intuition pump for this, remember that p=0.05 means "this result would happen by chance (only) 5% of the time even if the null hypothesis is true". So the more tests you run, the more likely it is that at least one of them will be a "significant" result just by chance—even if there's no effect there. See http://en.wikipedia.org/wiki/Multiple_comparisons and http://en.wikipedia.org/wiki/Post-hoc_analysis
$\begingroup$
$\endgroup$
I suggest you read the following:
Anderson, D.R., Burnham, K.P., Thompson, W.L., 2000. Null hypothesis testing: Problems, prevalence, and an alternative. J. Wildl. Manage. 64, 912-923. Gigerenzer, G., 2004. Mindless statistics. Journal of Socio-Economics 33, 587-606. Johnson, D.H., 1999. The Insignificance of Statistical Significance Testing. The Journal of Wildlife Management 63, 763-772.
Null hypotheses are rarely interesting in the sense that, from any experiment or set of observations, there are two outcomes: correctly rejecting the null or making a Type II error. The effect size is what you are probably interesting in determining and, once done, you should produce confidence intervals for that effect size.
