I am a statistical newbie. I recently started doing some text analysis in my text columns to find if a usage of specific word is causing my score to be high or low score. Previously I was finding the correlation then I came to know correlation does not tell you what variables affect other variables.

Q1: Is it possible to determine causality? and if yes what are the tools that I can use to do that? Im working in R and azure, so is there any tool I can use?

  • 3
    $\begingroup$ Proving causality is difficult. Usually, you need to design specific experiments: stats.stackexchange.com/q/2245/11849 $\endgroup$
    – Roland
    Dec 20 '17 at 7:52
  • 1
    $\begingroup$ E.g., the Stack Exchange people usually conduct manipulation experiments that show causality before changing something in their website: stackoverflow.blog/2017/10/17/… $\endgroup$
    – Roland
    Dec 20 '17 at 7:55
  • 2
    $\begingroup$ What @Roland said. Additionally, it is hard to define causaility exactly in a statistical setting. E.g. if you observe data sequentially (i.e., if you can assume that things at time $t$ cannot influence things at time $t-1$) you may be able work with the Granger Causality framework, which has little to do with what we would think about as causality in everyday life. Generally, the school of thought that you might be interested in checking out is called causal inference, see here: csm.lshtm.ac.uk/centre-themes/causal-inference $\endgroup$
    – Jeremias K
    Dec 20 '17 at 10:15

Causation is not in the data and cannot be. Data only contains correlation.

Most simply, if a variable $Y$ is correlated to $X$, $X$ can be seen as a cause of $Y$ if $X$ is controlled freely by the experimenter, which is done most often in a random way. This is not in the data but in the way the data was produced. Some frameworks were invented to broaden this limited definition, but the basic idea is still valid.

then I came to know correlation does not tell you what variables affect other variables.

Usually the word "affect" has an intuitive psychobiological meaning of "cause", while practically and mathematically this is still used about correlation. I'll try to explain.

Causation is often the way we represent correlation mentally (in our imagination). This is very useful to figure out what happens, and understand the structure of the data in a human way, and possibly to invent/correct/improve things. But strangely it's not in the data. What you can do instead is show how some correlations are explained by others: consequences. Note that the words "implied", "consequences", "explained" are all about correlation and not about causation.

Does the word "pissed off" causes a negative score (say measuring how much people are satisfied)? Actually, in my imagination I see it like this: anger both causes the word "pissed off" and a negative score. But this imagination can't be validated by the data, and I don't need it: I'll just validate "pissed off" is correlated with the score. I'll say it affects the score. Now what if I record anger itself by plugging some electrodes in people's brain and gut. I'll prove that the variable "anger" both explains the score "pissed off". But saying it is a "cause" is only in the way my imagination sees reality. Nothing in the data says that the variable "anger" is special as being the true first source of causality.

Unless you change the way the data is produced, introducing controlled intervention at some point or formalize what your imagination says about causality, you can't say anything about causality by looking at the data.


Not the answer you're looking for? Browse other questions tagged or ask your own question.