Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I recently read an article about how you can increase longevity by sleeping less. This article, like many others I've read, references a statistical study and implies that causation was found between two events(sleeping less and living longer). Well I'm not a statistician, but I'm aware of the common fallacies about causation and I find it very hard to accept the information given in this article and other similar ones. Even further, I can almost never know from an article if the original study found causation or not, without taking the time and digging into it.

Although I'd really like to, I'm not going to go into any depth bringing examples of how the causation might be wrong in the given article, because I would most likely be preaching to the choir. Luckily though, to my relief, the original article in NY Times only talks about correlation and even has a disclaimer in the end that basically says that the study doesn't imply causation.

Also, I now read that, in accordance with my intuition, it is actually really hard to find and/or test for causation. Reading through the wiki page on the subject I saw two models which seemed to be the most common ones used to find actual links in causation: Granger causality test and convergent cross mapping. But I see, that both of these have some gaps in them.

Now, I have a few questions regarding all this.

Q1: Are there any models that can, with high accuracy, find actual causation in such widespread studies? If so what are the most common ones?

Q2: How often do these widespread studies actually test for causation? E.g. if I read somewhere that a very broad study implies that A causes B, do I have any reason whatsoever to believe that there's something more than correlation behind those claims.

Q3: Is there a good technique that can be used by a non-statistician to filter out good statistics from the bad ones in everyday life.

share|improve this question
    
Regarding Q1: Are you only referring to observational studies? If not the easiest way to test for a causal effect is to conduct a experiment in which you e.g. apply food-restriction to one group of mice but not to another group and then test if the starving mice live longer. –  jokel Nov 5 '12 at 7:54
    
Well I'm referring to observational ones, yes. I understand, that blind (or even double-blind) experiments are quite good at finding/testing for causality. –  Deiwin Nov 5 '12 at 9:28
add comment

2 Answers

up vote 1 down vote accepted

Starting with Q2: convincing readers that a causal effect has been found is often the overriding concern of research articles in social sciences, psychology and medicine. However this usually cannot be done through a simple test but by proposing a causal theory, presenting evidence consistent with the theory in the form of correlations and other statistical patterns, pre-empting alternative explanations, and presenting evidence that is not consistent with the alternative explanations.

(Q1) Regression analysis allowing the researcher to examine the conditional correlation between an explanatory variable and a dependent variable while controlling for other possible explanatory variables is still probably the most common method. This does not by itself demonstrate causation between the variables of interest, but helps to isolate and reject competing causal explanations. In particular, it is difficult to argue that there is a causal relationship when there is no correlation after controlling for other variables (although when the explanatory variables are strongly correlated with each other, even this can be questioned).

Some other common methods that try to improve on basic regression analysis for eliminating competing explanations of hypothesised causal effects include

  • 'natural experiments', for instance where one region of a country has benefited from a new policy but another, otherwise similar, region has not
  • matching techniques such as propensity score matching, which try to control specifically for the fact that the 'treatment' group (e.g. people who sleep less) in an observational study may differ systematically from the 'control' group (people who sleep more) in ways that affect the outcome of interest (longevity)

(Q3) Arguably there is no simple technique for filtering out good statistics; it depends as much on knowledge of the field as knowledge of the statistical methods. Assuming an article shows a correlation between two variables, consider what the alternative explanations for that correlation could be (e.g. an unobserved variable influencing both, or reverse causation) and whether the article at least tries to eliminate those alternative explanations. There will nearly always be some form of regression or matching analysis to control for other variables; if nothing like that is mentioned in the abstract then a causal conclusion is likely to be unsound (but such articles would rarely get published in academic journals).

Also consider the level of generality claimed for the causal effect. E.g. the claim that less sleep leads to more longevity in abstract terms sounds like it is suggesting a universal effect, applicable to humans in all places and times. To make a general claim like that plausible would depend on subject knowledge (e.g. prior evidence that the determinants of longevity are similar across countries) and a strong argument that the sample is representative (e.g. the article describes how they were selected and discusses how potential sample bias was avoided).

For me the most convincing evidence from observational studies is when a series of studies by different researchers, using different datasets and techniques, have consistently found evidence that does not contradict the proffered causal explanation, and progressively eliminated alternative explanations.

share|improve this answer
    
Thanks! Really helpful. –  Deiwin Nov 5 '12 at 12:44
add comment

Q2: As far as I know there are 2 good ways to test for causality. It is done by the design of the study. Two designs work here:

  • In a situation where you can control ALL independent variables (usually in experiments). This makes sure that A is causing B, and there is no C causing the effect (and also not the reverse effect; B causing A).
  • In longitudinal studies.

Q1: Rubin's Causal Model: http://en.wikipedia.org/wiki/Rubin_causal_model

Q3: There is no "trick" to seeing whether the statistics of one single article are good. I find the most important questions you should ask yourself:

  • Who did the research?
  • What is to be gained by the researcher with a favorable outcome?
share|improve this answer
    
On bullet 1: Controlling all independent variable can also be wrong. Simple example: B is caused by A and C, but A does not cause C. Now, if you control for B and you'll be wrong about the causal impact of A on C (you'll think there is one). Also, in experiments that randomise and control, it is the randomisation that warrants the causal inference; control 'just' makes the inference more precise. –  conjugateprior Nov 5 '12 at 12:45
1  
You might find Winship and Morgan's overview article helpful. –  conjugateprior Nov 5 '12 at 12:53
    
Controlling all other independent variables, while randomizing the variable expected to have a causal relation? –  PascalvKooten Nov 5 '12 at 12:57
    
Even then. Try it: assume A and C are normalised test scores unrelated in the population: dd <- data.frame(A=rnorm(100), C=rnorm(100)) # A is randomised and unrelated to C. Now make B, say college admission, depend on them: transform(dd, B=ifelse(A+C > 1, 1, 0)) ## score over 1 get in. The correct causal inference is given by not conditioning on B: summary(lm(C ~ A, data=dd)) # no effect. Conditioning on B: summary(lm(C ~ A + B, data=dd)) makes it look like A causes C to be go down, but it doesn't (they're unrelated by construction). This is not specific to regression, btw. –  conjugateprior Nov 5 '12 at 13:19
    
You might be thinking 'but wait, that B really is a dependent variable', and you'd be right, in a sense -- this effect is closely related to the hazards of 'selecting on the dependent variable'. In this case it's just not your dependent variable (that would be C in the example.) Nevertheless, if you didn't know A and C caused B, or more generally you didn't know that B had A and C somewhere way back in its causal chain, then you'd list it as an independent variable, condition on / control for it and things would go wrong, as they do above. –  conjugateprior Nov 5 '12 at 13:27
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.