Suppose I want to determine the effect of an intervention on a health score. I do not have the means to design an experiment to randomly assign people to the control and treatment groups. The way I recruit people to apply for the intervention is by mailing advertisements to their homes, or going to hospitals and giving seminars about the intervention. After a month, I want to measure the mean health score for people who signed up for the intervention vs. the rest of the population see if there is a statistically significant difference. How would you apply the standard ANOVA/t tests if the samples were not random?
2
-
1$\begingroup$ I see nothing wrong with doing that as a way to describe the effect of the intervention. But I don't think there is any inference procedure that you could use to claims a statistically significant result even if it seems to be a large effect. $\endgroup$– Michael R. ChernickCommented Dec 27, 2016 at 19:26
-
$\begingroup$ Is there some reason why you can't do both a pre-intervention and a post-intervention "health score" on the individuals who sign up? $\endgroup$– EdMCommented Dec 27, 2016 at 20:34
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
You apply the standard test that you would have applied had you randomized. Statistical significance tells you if you have evidence that the groups differ in the population on the outcome. Then you have to interpret that result.
There is a whole literature on causal inference. The Wikipedia entry on causal inference starts: "Causal inference is the process of drawing a conclusion about a causal connection based on the conditions of the occurrence of an effect." Currently, you have the condition of the occurrence of an effect - you want to know if that difference has occurred because of your intervention - i.e. is the effect causal.
Determining whether a relationship is causal is a much bigger question than can be answered on a forum like this - they have been being debated for more than 50 years, books have been written on the subject, and other books have been written that disagree with those books. There's a whole journal devoted to the subject: https://www.degruyter.com/view/j/jci
However, the way that I sometimes think of it is to say:
- I have found an effect.
- Maybe it's causal, or maybe there's another mechanism. Can I eliminate this other mechanism. If so, try.
- Is there another potential mechanism? If so, return to 2.
- The more alternative mechanisms I've eliminated (and I can never eliminate them all, because I haven't thought of some of them), the more confident I am that I have found a causal relationship.
That's something of an oversimplification - some of the references in the Wikipedia entry are good starting points. If you make some progress with these, you'll probably be able to come back with a more specific question.
answered Dec 27, 2016 at 19:23
-
$\begingroup$ I think you miss the point. The OP has to first show that the effect is statistical meaningful before picking a methodology to show that the intervention caused the effect. Take the classic case where you want to find out if two variables are correlated. You would first take a sample and compute a sample correlation and test the hypothesis that the true correlation is different from zero. Suppose you find significant correlation and even observe a high correlation between X and Y. You would still need additional information to say that a change in X caused a change in Y. $\endgroup$ Commented Dec 27, 2016 at 19:43
-
$\begingroup$ Although the methods of Rubin and Pearl allow for causality to be shown in observational studies certain assumptions need to be made. The methods involve such things as structural equation modeling, path analysis and counterfactuals.. $\endgroup$ Commented Dec 27, 2016 at 20:17
-
$\begingroup$ As I reread your post I see that you admit that you are oversimplifying. Your discussion is a good tutorial about the issues. It can be applied to the OPs question where the study is observational. But I don't completely understand it and although I think that although much of the theory and methods are becoming accepted by the statistical community there may be some aspects that are not. $\endgroup$ Commented Dec 27, 2016 at 20:23
-
1$\begingroup$ I think that the question OP asked doesn't make sense as it was asked, rather than admonish a new user, I tried to clarify what they might have better asked about. $\endgroup$ Commented Dec 27, 2016 at 21:15
-
$\begingroup$ I don't think they were struggling with running the test - they asked if they should apply the standard anova/t-test. $\endgroup$ Commented Dec 27, 2016 at 21:17
$\begingroup$
$\endgroup$
An alternative answer is that a simple ANOVA/ANCOVA adjusting for pre-intervention score/t-test will not tell you whether the intervention does anything, because people that use it may differ from those that do not for reasons other than the intervention. Very often such differences are bigger than the intervention effect, if any.
Attempts to deal with that include things like analyses stratified by propensity score - a score/probability for choosing the intervention estimated using all information that affects the decision (if you do not have some of that information that puts the analysis in question) - and other similar analysis approaches.
