Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I apologize if this question has been done to death, but as a non-statistician I really don't know what is the bottom line takeaway. I am looking at a sample of 30,000 individuals who were the subject of an economic intervention. This intervention produces a mean increase in the annual income of the subjects of $2000. However the result is not statistically significant at usual levels. From a statistical viewpoint, has the intervention failed, and if not what further can be done?

share|improve this question
8  
How did you test the increase for significance? Was a control group used? –  Nick Stauner Jun 1 at 19:48
    
The love the strict adherence to the ideals of the scientific method implicit in the question. Lies, damn lies and statistics right? :D –  Nathan Cooper Jun 2 at 16:02
    
@NathanCooper: More like strict adherence to the conventions of statistically naive scientific practice. If statistics is to blame, its fault is in providing simple methods with too much appeal in even inappropriate circumstances, or failing to emphasize their limitations sufficiently. (The probability that statistics is to blame is pretty low IMO.) –  Nick Stauner Jun 2 at 16:47

3 Answers 3

Statistical insignificance does not mean that the effect being tested for does not exist, but rather, that the data that was observed does not furnish strong evidence for the existence of that effect.

For example, if you have an unloaded six-sided die, but the numbers on its faces are {1,2,3,4,5,5} instead of {1,2,3,4,5,6}, and you roll it only 3 times, it may not be evident through such a small sample size that the die would give you more fives than ones. That doesn't mean the die isn't different than a normal die (after all, we have the benefit of inspecting it and we can clearly see it is different)--it may simply be that we need to collect more data about the die's observed behavior in order to make a statistically significant inference about the intrinsic properties of the die.

Analogously, it may be that even a sample size of 30000 may not be sufficient to detect a difference in the behavior of your population under two treatments, because your statistical test has low power. Or, maybe the truth is that the mean increase you're observing is actually due to random chance and no effect truly exists. Since you have not specified your tolerance for Type I error, I can't really speak to that.

The takeaway here is that failure to detect significance doesn't mean no effect exists--it simply means that, by random chance or by lack of power, the data furnishes insufficient evidence to claim that the hypothesized effect exists with a high degree of confidence.

share|improve this answer

Well, this is certainly not good news. Sorry.

Your results don not provide any evidence for the existence of an effect. The effect, of course, might still exist: it could be smaller or more variable than you expected, or your experiment was somehow flawed and failed to detect it.

So, what can you do now?

0) Check your data. Make sure nothing silly has happened. Missing values sometimes get coded as 0s/-1s/99s, and these numbers obviously shouldn't be entered into your analysis as actual values. Similarly, if you're randomizing people to treatments/controls, make sure these groups are actually similar. People get bitten by these sorts of bugs all the time.

1) Perform a power analysis. Ideally, you would have performed one before beginning the project, but doing one now can still help you determine whether your experiment, as performed, would have a reasonable chance of detecting your expected effect. If not, (perhaps your drop-out/noncompliance rate was very high), you might want to perform a larger experiment.

You should not add subjects, run the analysis, and repeat until your result becomes significant, but there are lots of strategies for mitigating the problems associated with taking multiple "looks" at your data.

2) Look at sub-groups and covariates. Perhaps your proposed intervention works best in a specific geographical region, or for younger families, or whatever. In general, it would best to specify all of these comparisons ahead of time, since exploiting "experimenter degrees of freedom" can dramatically increase the false positive rate.

That said, there's nothing wrong with looking per se. You just need to be upfront about the fact that these are post-hoc/exploratory analyses, and provide weaker evidence than an explicitly confirmatory study. Obviously, it helps a lot if you can identify plausible reasons for why the subgroups differ. If you find a hugely significant effect in the North, but nothing in the drought-stricken, war-ravaged South, then you are in pretty good shape. On the other hand, I'd be a lot more skeptical about a claim that it works on subgroups of people born during full moons but only at high tide :-)

Good luck!

share|improve this answer
    
2) I feel someone could grab the wrong end of the stick here. If you go on a fishing expedition you lose pretty much all the evidential power. It is possible to check for subgroups in the upfront design, but there are usually enough sensibly distinct groups to provide bogus type I's in this kind of exploratory analysis. –  Nathan Cooper Jun 2 at 16:00
1  
@NathanCooper, you're absolutely correct. The paper I linked to describes how bad fishing expeditions can get (and it's pretty grim). However, having spent a lot of time and money gathering data, it would be nice to get something out of it, even if that something is tentative hypothesis that needs to be rigorously confirmed. If including very plausible covariates produces a big effect size, then that can be relatively convincing (but yes, still needs to be explicitly confirmed). If one needs a crazy-quilt of inclusions and exclusions to push something just into significance, then...no. –  Matt Krause Jun 2 at 16:36

Regarding the title question: Categorically, no. In your case, not enough info, hence my comment and downvote. Also, IMO, questions that conflate statistical and practical significance have been done half-to-death here, and you haven't said enough to make your question unique. Please edit; I'll undo my downvote if I see improvement (it's locked now), and probably upvote if it's substantial. Your question addresses a common, important misconception that deserves being done the rest of the way to death, but as is, it's hard to say anything new about your situation that would make it a useful example.

From a statistical viewpoint, has the intervention failed, and if not what further can be done?

Again, what have you done so far? It's also quite possible that your analysis has failed, to borrow your term (IMO, "failed" is clearly too harsh in both cases). This is why I asked about your test. There's a fair amount of controversy surrounding pre-post analysis options, and random sampling or lack thereof is relevant to the choice of analytic options (see "Best practice when analysing pre-post treatment-control designs"). This is why I asked about a control group.

If your choice of test can be improved, do that (obviously). In addition to checking your data (as @MattKrause wisely suggested), check your test's assumptions. There are quite a few involved in the usual pre-post designs, and they're violated often.

  • Normal distributions are likely to be poor models, especially for change scores and financial data. Consider nonparametric analyses.
  • Heteroskedasticity is common, especially without random selection or with a partially stochastic intervention. Some tests are more sensitive to this than others – especially the conventional ones.
  • Conventional ANCOVA assumes no interaction between interventions and covariates. If baseline income affects the viability of the intervention, you should probably use moderated regression instead $(\text{Final Income = Baseline Income + Intervention? + Interaction + Error}$, basically), assuming you do have a control group. If you don't, do you have more than 2 times?

What other info about your individuals do you have? Exploring covariates and moderators is a good way to reduce the amount of statistical "noise" (error) your intervention's "signal" (effect) has to overwhelm for your test to "detect" it (support rejection of the null). If you can explain a lot of variance by means other than your intervention, or explain why your intervention doesn't affect everyone equally, you might get a better sense of how big your intervention's effect really is, all else being equal – which is rarely the default state of nature. I believe this was the spirit of Matt's suggestion #2.

Regarding his caveat, don't be afraid to explore covariates and moderators you haven't specified in advance; just adopt an exploratory mindset and acknowledge this epistemological transition explicitly in any report you publish. The crucial point that bears repeating about statistical and practical significance is that their overlap is generally limited. Much of the practical significance of statistical significance is in what you intend to make of it. If you're seeking evidence to support further research (e.g., for a research grant), rejection of exploratory hypotheses may be enough. AFAIK, this is the only kind of practical significance that statistical significance is supposed to imply by default, and explains the choice of terminology historically: significant enough to justify more research.

If you're looking for a statistical viewpoint on whether your intervention is worthwhile, you're probably asking in the wrong way. Statistical significance is not intended to answer this by itself; it only directly represents an answer to a very specific question about a null hypothesis. I suppose this amounts to another suggestion: check your null hypothesis. It usually defaults to stating that the effect observed in your sample is due entirely to sampling error (i.e., effect of intervention = 0). Are you really interested in any change whatsoever? How consistent do you need it to be to justify the intervention? These questions partly decide the appropriate null; you need to answer them.

In confirmatory testing, you need to answer in advance. Since you've already run a test, any new tests of the same kind with different null hypotheses but the same sample would be exploratory. Unless you can collect another sample, it would probably be best to regard other kinds of tests as exploratory too. The strict sense of confirmatory hypothesis testing is particularly strict about the "no peeking" rule; IMO, this is a weakness of the hypothesis-testing paradigm as a whole. AFAIK, Bayesian analysis can be a little less strict about this, and might benefit you particularly if you can collect more data, because your current result could help inform your prior probability distribution.

Another way to approach the issue is by focusing on effect size and your confidence interval. $2K is a change in the direction you wanted, right? If your test's results meant what I think you think they meant, then there's a better than 5% chance you'd find a negative change if you were to repeat the study, assuming the intervention had no effect. If your investment had any positive effect at all, the probability is lower than your p value. If you're invested heavily enough in the prospect of the treatment, maybe you should replicate the study. Again, you know better than I what else affects that decision.


P.S. Despite my intro, I've managed to say plenty about this "half-dead" topic. Hopefully I've provided a useful summary of ideas other than those in preexisting answers, but I wouldn't be surprised if much of it isn't very useful to you personally. A big reason I wanted more info is that answering a vague question well practically necessitates covering a lot of unnecessary bases, which is kind of a waste of time. Nonetheless, if you grace us with an edit, I'll probably subsection off whatever no longer applies, and I might expand on what still does. It's evident from the incoming views that the question resonates with the audience here, so this could become a very useful question with a little more work.

share|improve this answer
    
+1 for discussing the importance of practical vs. statistical significance (and thanks for the shout-outs/elaborations!) –  Matt Krause Jun 2 at 12:34
    
I think the reason this question is done to death is that, for the layman, there is simply no one sentence take-away. This is very unnerving because if the intervention is, say, a new drug, knowing its efficacy is literally a matter of life and death. –  tom russell Jun 5 at 16:05
    
@tomrussell: one-sentence takeaways...To the title question: no, statistical insignificance could easily represent a failure to control other overwhelming sources of error rather than a conclusion of zero effect. To the question of failure: no, from a statistical standpoint, the study is inconclusive. Statistics can make one-sentence conclusions in applications more informative than they would otherwise be, but proper statistical practice certainly does take more patience to understand than that. People without the attention span for several sentences should consider ethics of misinformation. –  Nick Stauner Jun 5 at 16:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.