Suppose I thought that ingesting greater than 100 mg of chemical X annually noticeably decreased one's weight. Also, I had data (from a "natural" experiment) from 100 people (some male and some female) measuring how much of chemical X they had eaten, and their weights at the time of the experiment.

  • What is the best way to test whether or not eating said amount of X leads to weight loss?
  • 1
    $\begingroup$ How did the subjects end up consuming different quantities of the chemical? Calling something a "natural experiment" implies that there is an exogenous source of variation in the treatment variable. Do you have reason to believe that true in this case? $\endgroup$ – Michael Bishop May 24 '11 at 22:33

By a "natural" experiment you mean that you do not control, by randomization, say, the amount of chemical X that each subject takes. This is also often called an observational study. Do you know the difficulties in drawing conclusions about cause and effect from such data?

It's really not a question about statistical tests - they are blind to causality. You can use standard methods to test if intake of chemical X is associated with weight loss correcting for gender if you like, but that does not in itself prove that intake of X causes weight loss. An association might be viewed as evidence in the direction of a causal effect, but how strong the evidence is, and how serious it will be taken, is much more a question of understanding the subject matter than the statistical test.

There is a literature on causal inference, which I am only superficially familiar with, which gives a more nuanced picture on what you can say about causal effects and how you can do it, but a basic premise is a set of untestable assumptions.

If you can provide more details about what you know and what you want, I might be able to give you some appropriate references. There are also other, related, questions with answers here and here.


It sounds like everyone in your sample has ingested X. If your hypothesis is that ingesting X causes you to lose weight, you need a sample of people, some of whom have ingested X and some of whom haven't. If your hypothesis is that the more X you ingest, the more weight you lose, then, of course, it's okay if your entire sample has ingested X. You can also split up your sample into two groups if you thought that ONLY those who ingested > 100mg will lose weight, but there are a lot of potential problems there and I wouldn't recommend it.

From the sound of it, what you probably want to start with is a simple regression to see if there is any connection between X and weight. You can simply graph your sample, with weight lost (or gained) on one axis and amount of X ingested on the other. Using a stat package like SPSS or R will tell you the correlation between X and losing weight (r), and how much of the variance (of weight loss) is accounted for by X (called r^2). It's always a good idea to have a control group, though, to make sure the weight loss could not be attributed to something else.

There are other things you can do depending on what data you have, but this is a good place to start I think.


As far as your statistical test, it might be a choice between 1) ancova with pretest weight as the covariate and 2) anova with change scores as the outcome. You'd use ancova if you believed posttest weight would naturally be different from pretest weight even without the treatment, and that posttest weight would be a linear function of pretest weight with a correlation of at least .3. You'd use anova on change scores if you believed that absent the treatment there would be no expected change in mean weight. Cook and Campbell's Quasi-Experimentation has a particularly thought-provoking section on these issues.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.