When does correlation matter when there is no theoretical relationship?

Question

Summary: Correlations does not imply causation has became mainstream. I am interested in understanding what is required for the implication to actually take place.

Detailed question:

1. Simple case: when I have a relationship derived from theory (say, the gravitational force $F(m) = m \times g$) I can make measurements and two things can happen:

   • either the data (together with its uncertainty and/or statistical dispersion) fits the theoretical curve = the theory is good enough under current measurement capacities
   • or it does not = the theory is missing something

2. Complicated case: how does this approach works when there is no underlying, theoretically derived, relationship?

I am specifically interested in a few conclusions from the DowJones Women at the Wheel report, such as

The overall median proportion of female executives is 7.1% at successful companies and 3.1% at unsuccessful companies, demonstrating the value that having more females can potentially bring to a management team.

How to handle such results? What is missing in order to make (or not) the link? (the conclusion above is IMHO vaporous at best).

In other words: what kind of data is needed, and which analysis must be done on it to be able to reasonably claim that there is a relationship of some kind? (again, in the case there is no theoretical formula handy)

Answer 1

This is such a loaded question and much too complex to answer adequately in a single response. That said, your intuition that the DowJones report is lacking in causal evidence is correct.

Some of the previous responses noted that causal evidence could come from an RCT, an IV, a Difference-in-Difference, etc. However, all of these really boil down to one thing: the ability to explain away other possible causes or reasons for the correlation. For example, in the report you describe, it is quite easy to think of a number of reasons for the correlation between success and number of women executives apart from "women executives cause success." In the end, if you can think of a plausible alternative cause, then more causal evidence is likely needed.

One final thought: just because there is no causal evidence does not mean that the relationship is NOT causal. In the DowJones report, they do not provide causal evidence, but that does not necessarily mean that women executives do not cause success.

Answer 2

Experimental data would be a theoretically appealing, but practically difficult solution:

Randomly assign proportions of female executives to companies and see which are successful and which aren't (in terms of a useful metric, like return on investment), for example by comparing the mean returns of firms with "many" women to those with "few" women. Or do a regression of returns on the proportion of women.

That way, you ensure that there is no bias that might occur in observational studies when other important factors related to female proportion cannot be controlled for ("omitted variable bias", "confounding"). Such bias might arise if, for example ("endogeneity"), women disproportionately happened to work in more profitable sectors (which I doubt is the case, but which could invalidate a causal claim). Or reverse causation: firms that are already successful might choose to hire more women (again, I am not saying this is what is going on).

Obviously, this strategy will be difficult to put in practice, to put it mildly. My answer is only an attempt to highlight what the "gold standard" of experimental data, as it is sometimes called, would amount to in this case.

A more practically promising strategy might be to look for an "instrumental variable". See e.g. here.

Answer 3

It's like the difference between theory and praxis, identifying causal relationships vs cultivating observational information. One could argue that, in terms of the history of science, the former is a Popperian stance while the latter is a Baconian framework. A Popperian theory is one that is fundamentally driven by theoretical and mathematical assumptions, e.g., as in physics. Baconian science, on the other hand, is primarily observational in nature and attempts to develop theory from the cumulation of initially purely observational information. The biological sciences or the modern field of informatics are representative of this latter point of view.

The literature on causality and the power of experimental design and random controlled trials (RCTs) vs less powerful observational studies that may be randomly sampled and nationally projectable but are not otherwise considered experimental designs is huge and quite contentious. Judea Pearl is probably the most prominent scholar on the "causal" side as his work is widely read and cited but others include James Heckman (the Nobel Laureate) and Donald Rubin (prominent Harvard Bayesian). Wrt leveraging weaker observational studies for causal relationships, it seems that everyone has a point of view about this but there is little in the way of consensus that I'm aware of. The "causal" guys tend to be kind of rigidly orthodox in their insistence on RCTs as the gold standard for all scientific endeavor while the observational guys tend to sort of cower under this withering glare as "nonscientists."

So, are you interested in determining causal relationships? Are the empirical results that are driving the observed correlation an artifact of the experimental data or does your "theory" need to be adjusted? In other words and to Christoph Hanck's point, is the relationship captured by this correlation an artifact of some statistical endogeneity (variously defined but Wooldridge's definition is best and simply refers to undesired association between the model residual and the predictors or X variables). If so, then economists' would prescribe developing an instrumental variable or set of IVs to control for this endogeneity. IV's are a big subject that's too broad for this little post.

If your study or project does not require you to make rigorous statements and judgments about causality, then you are on more observational ground where determining associations among the information is the rule and causality, depending on your POV, takes a back seat.

It sounds like the study you reference on "Dow Jones Women at the Wheel" properly falls in the observational category. Therefore, it is on much weaker interpretive ground. Specifically to your point, and without even reading the report, the extracted quote that the observed difference in proportions, "demonstrat(es) the value that having more females can potentially bring to a management team" is not substantiated with any supporting evidence. One is left wondering whether or not this difference matters. Unfortunately and based solely on the quote, there is nothing to support this conclusion beyond eyeballing the percentages. For some people, this includes many marketers as well as those with a political or ideological bias towards confirmation of one opinion or another, this visual "evidence" is enough. There may be additional information in the report that goes further in substantiating this claim, but it's not evident from the quote. As I indicated, I haven't drilled into the report to confirm or disconfirm this and I don't want to.

This gets to the final point worth making and has to do with the intended audience for this analysis. The study you cite falls into the domain of market research. Market researchers are way downstream both in reputation and importance from the vaunted frameworks of Popperian vs Baconian science. The biggest problem is that it is a field populated to a huge degree by the technically illiterate and innumerate who shrink at the use of geek jargon (even words like "correlation" can be suspect) and make every effort to scrub their reports of verbiage which the lowest common denominator of unskill cannot sustain. In large part, this is due to the emphasis from the c-suite level on down in many companies on promoting "great communicators," people who can orate effectively but who amount to empty suits. For confirmation of this just consider the total market caps of technically literate companies (e.g., Google, Amazon, Apple and FB, about 800 billion dollars) vs the market caps of the large agency holding companies (technically illiterate, e.g., WPP, Publicis, Omnicon, Interbrand, about 80 billion dollars). The literate entities have a 10x advantage in valuation over the illiterate ones. In other words, the smoke and mirror agencies aren't fooling anyone, least of all Wall St.

That said, this doesn't mean that all marketers are sleazebags who are only in the business to blow smoke and sell mirrors. There are those in this business who are highly technically strong and rigorous in their methodology and reportage. It's just that they are regrettably in the minority.

Answer 4

As the respondents above note, causal inference (which seems to be what you're going for) is tricky. One approach is the experimental one, discussed above. But this can be difficult to pull off if you can't control the situation or there is little theoretical understanding to help you identify, for example, a good instrument.

One other way to go is to think about the mechanisms that could explain this relationship--what intervening variables or processes (depending on your definition) explain this apparent effect? If you can identify them based on existing theory, you could conduct a follow-on study to see if those mechanisms are in place. For example, if you observe X leading to an increase in Y, and you think this is because X makes Z more likely, which has a strong influence on Y, then you could look for evidence of Z in cases of X.

This book might be useful-https://mitpress.mit.edu/index.php?q=books/case-studies-and-theory-development-social-sciences. It's focused on qualitative work more than quantitative, but some of the high-level discussion on causation is applicable to stats.