I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is there anything else needed to be included in my report?
Hypothesis testing versus parameter estimation
Typically, hypotheses are framed in a binary way. I'll put directional hypotheses to one side, as they don't change the issue much. It is common, at least in psychology, to talk about hypotheses such as: the difference between group means is or is not zero; the correlation is or is not zero; the regression coefficient is or is not zero; the r-square is or is not zero. In all these cases, there is a null hypothesis of no effect, and an alternative hypothesis of an effect.
This binary thinking is generally not what we are most interested in. Once you think about your research question, you will almost always find that you are actually interested in estimating parameters. You are interested in the actual difference between group means, or the size of the correlation, or the size of the regression coefficient, or the amount of variance explained.
Of course, when we get a sample of data, the sample estimate of a parameter is not the same as the population parameter. So we need a way of quantifying our uncertainty about what the value of the parameter might be. From a frequentist perspective, confidence intervals provide a means of doing, although Bayesian purists might argue that they don't strictly permit the inference you might want to make. From a Bayesian perspective, credible intervals on posterior densities provide a more direct means of quantifying your uncertainty about the value of a population parameter.
Parameters / effect sizes
Moving away from the binary hypothesis testing approach forces you to think in a continuous way. For example, what size difference in group means would be theoretically interesting? How would you map difference between group means onto subjective language or practical implications? Standardised measures of effect along with contextual norms are one way of building a language for quantifying what different parameter values mean. Such measures are often labelled "effect sizes" (e.g., Cohen's d, r, $R^2$, etc.). However, it is perfectly reasonable, and often preferable, to talk about the importance of an effect using unstandardised measures (e.g., the difference in group means on meaningful unstandardised variables such as income levels, life expectancy, etc.).
There's a huge literature in psychology (and other fields) critiquing a focus on p-values, null hypothesis significance testing, and so on (see this Google Scholar search). This literature often recommends reporting effect sizes with confidence intervals as a resolution (e.g., APA Task force by Wilkinson, 1999).
Steps for moving away from binary hypothesis testing
If you are thinking about adopting this thinking, I think there are progressively more sophisticated approaches you can take:
- Approach 1a. Report the point estimate of your sample effect (e.g., group mean differences) in both raw and standardised terms. When you report your results discuss what such a magnitude would mean for theory and practice.
- Approach 1b. Add to 1a, at least at a very basic level, some sense of the uncertainty around your parameter estimate based on your sample size.
- Approach 2. Also report confidence intervals on effect sizes and incorporate this uncertainty into your thinking about the plausible values of the parameter of interest.
- Approach 3. Report Bayesian credible intervals, and examine the implications of various assumptions on that credible interval, such as choice of prior, the data generating process implied by your model, and so on.
Among many possible references, you'll see Andrew Gelman talk a lot about these issues on his blog and in his research.
- Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological methods, 5(2), 241.
- Wilkinson, L. (1999). Statistical methods in psychology journals: guidelines and explanations. American psychologist, 54(8), 594. PDF
Just to add to the existing answers (which are great, by the way). It is important to be aware that statistical significance is a function of sample size.
When you get more and more data, you can find statistically significant differences wherever you look. When the amount of data is huge, even the tiniest effects can lead to statistical significance. This does not imply said effects are meaningful in any practical way.
When testing for differences, $p$-values alone are not enough because the required effect size to produce a statistically significant result decreases with increasing sample size. In practice, the actual question is usually whether there is an effect of a given minimal size (to be relevant). When samples become very large, $p$-values become close to meaningless in answering the actual question.
If there was a reasonable basis for suspecting your hypothesis might be true before you ran your study; and you ran a good study (e.g., you didn't induce any confounds); and your results were consistent with your hypothesis and statistically significant; then I think you are fine, as far as that goes.
However, you shouldn't think that significance is all that is important in your results. First, you should look at the effect size as well (see my answer here: Effect size as the hypothesis for significance testing). You might also want to explore your data a bit and see if you can find any potentially interesting surprises that might be worth following up on.
Before reporting this and this and this and this, start by formulating what do you want to learn from you experimental data. The main problem with usual hypothesis tests (these tests we learn at school...) is not the binarity: the main problem is that these are tests for hypotheses which are not hypotheses of interest. See slide 13 here (download the pdf to appreciate the animations). About effect sizes, there's no general definition of this notion. Frankly I would not recommend to use this for non-expert statisticians, these are technical, not natural, measures of "effect". Your hypothesis of interest should be formulated in terms understandable by the laymen.
I'm far from an expert on statistics, but one thing that has been emphasised in the stats courses I have done to date is the issue of "practical significance". I believe this alludes to what what Jeromy and gung are talking about when referring to "effect size".
We had an example in class of a 12 week diet that had statistically significant weight loss results, but the 95% confidence interval showed a mean weight loss of between 0.2 and 1.2 kg (OK, data was probably made up but it illustrates a point). While "statistically significantly"" different from zero, is a 200gram weight loss over 12 weeks a "practically significant" result to an overweight person trying to get healthy?
This is impossible to answer accurately without knowing more details of your study and the person's criticism. But here's one possibility: if you've run multiple tests, and you choose to focus on the one that came out at
p<0.05 and ignore others, then that "significance" has been diluted by the fact of your selective attention to it. As an intuition pump for this, remember that
p=0.05 means "this result would happen by chance (only) 5% of the time even if the null hypothesis is true". So the more tests you run, the more likely it is that at least one of them will be a "significant" result just by chance—even if there's no effect there. See http://en.wikipedia.org/wiki/Multiple_comparisons and http://en.wikipedia.org/wiki/Post-hoc_analysis
I suggest you read the following:
Anderson, D.R., Burnham, K.P., Thompson, W.L., 2000. Null hypothesis testing: Problems, prevalence, and an alternative. J. Wildl. Manage. 64, 912-923. Gigerenzer, G., 2004. Mindless statistics. Journal of Socio-Economics 33, 587-606. Johnson, D.H., 1999. The Insignificance of Statistical Significance Testing. The Journal of Wildlife Management 63, 763-772.
Null hypotheses are rarely interesting in the sense that, from any experiment or set of observations, there are two outcomes: correctly rejecting the null or making a Type II error. The effect size is what you are probably interesting in determining and, once done, you should produce confidence intervals for that effect size.