21
$\begingroup$

Context

A popular question on this site is " What are common statistical sins?". One of the sins mentioned is assuming that "correlation implies causation..." link

Then, in the comments with 5 upvotes it is suggested that: "Google makes $65B a year not caring about the difference."

At the risk of over-analysing a light quip, I thought this might be a useful discussion point for fleshing out the distinction between correlation and causation and the practical relevance of the distinction; and perhaps it could highlight something about the relationship between machine learning and the distinction between correlation and causation.

I'm assuming the comment is addressing technologies that underlie the generation of search engine results and advertising display related technologies.

Question

  • To what extent is the distinction between correlation and causation relevant to Google's income generation, perhaps focusing particularly on the generation of income through advertising display related technologies and quality search results?
Improve this question
$\endgroup$
2

4 Answers 4

Reset to default
13
$\begingroup$

The simple answer is that Google (or anyone) should care about the distinction to the extent that they intend to intervene. Causal knowledge tells you about the effects of interventions (actions) in a given domain.

If, for example, Google wishes to increase click-through rates on ads, increase the number of users of GMail or Google+, or induce users to use Google rather than Bing, then they need to know the effects of potential actions (e.g., increasing the font size of ads, promoting Google+ in print magazines, or publicizing differences between Google and Bing search results, respectively). Correlation is good enough to make Google's search engine work well, but for their other systems (and their business overall) the distinction often matters.

It is worth noting that Google (and many firms with web-based businesses) are constantly running online experiments. This is of the simplest and best ways to identify and estimate causal dependencies.

Improve this answer
$\endgroup$
9
  • $\begingroup$ (+1) as long as the predictors a) correlate and b) allow to predict future outcomes correctly, one should not care about causation. $\endgroup$
    – steffen
    Commented Aug 10, 2011 at 14:18
  • 2
    $\begingroup$ We are entering an era of a revival of experimental research in behavioral sciences. In the 1950s, almost all of statitsics was experimental research, with applications in agriculture. But around 1980s, people have recognized these techniques are not of much help with observational data, which is all you can do in most social sciences. Now, at least in the niches of online marketing research, if you are Amazon or Google or Bing, you can run experiments and get the cleanest form of causal inference possible. $\endgroup$
    – StasK
    Commented Aug 10, 2011 at 14:32
  • $\begingroup$ @StasK, Considering the sample sizes they're likely dealing with even a "short" experiment they are likely to produce highly useful results. What a treasure trove that must be. $\endgroup$
    – Brandon Bertelsen
    Commented Aug 10, 2011 at 20:41
  • $\begingroup$ It's interesting to note that Google's "groups" facility is really, really poor. It's kind of like they built a group discussion facility that correlates with being good, but they haven't figured out what causes a group discussion facility to be good. But this is a common problem in marketing -- all too often features of competing products are copied without understanding the underlying motivation for the features. $\endgroup$
    – Daniel R Hicks
    Commented Aug 11, 2011 at 11:59
  • 1
    $\begingroup$ @StasK: The real world is probably not as ideal as you pretend here. I agree that experiments are a great tool to do causal inference. However, experiments raise specific problems as well Sometimes it might be better to make causal inferences with an observational study rather than with an experiment. One criticism might be in what sense results from a controlled experiments might be generalized to a "real live" setting. Some authors call this "external validity". $\endgroup$
    – user5644
    Commented Aug 11, 2011 at 12:10
6
$\begingroup$

First, it is just a quip and is incorrect. Google has a lot of very talented statisticians, information retrieval experts, linguists, economists, some psychologists, and others. These folks spend a lot of time educating a lot of non-statisticians about the difference between correlation and causation. Given that it's a large organization, there may be pockets, even big pockets, of ignorance, but the assertion is definitely false. Moreover, a lot of that education faces customers, especially advertisers.

Deeper answer: The difference is extremely important. Just look at search results ranking, and allow me to extend beyond just "correlation" to include measures of similarity, scoring functions, etc. Some pages are measured to be good results for certain queries. These have a variety of predictor features that are important to their ranking. In contrast to these good pages that are good results for queries is a set of webpages that are pages that are very bad results for the same queries. However, creators of those pages spend a lot of effort to make them look like good pages from a numerical point of view, such as text matches, internet linkage, and more. However, just because these pages are numerically "similar" to good pages doesn't mean that these are, in fact, good pages. Therefore, Google has invested and will continue to invest a lot of effort determining what reasonable features distinguish (separate) good and bad pages.

This isn't quite correlation and causation, but it's deeper than that. Good pages for certain queries may map into a numerical space where they appear similar and distinct from many irrelevant or bad pages, but just because results are in the same region of the feature space does not imply they come from the same "high quality" subset of the web.

Simpler answer: A very simple perspective is to address the ranking of the results. The best result should be first, but just because something is ranked first doesn't mean that it's the best result. By some metrics of scoring, you may find that Google's ranking is correlated to a golden standard of quality assessments, but that doesn't mean that their ranking implies that the results are truly in this order in terms of quality and relevance.

Update (third answer): Over time, there is another aspect that affects all of us: it is that the top Google result may be deemed authoritative, because it is the top result on Google. Although link analysis (e.g. "PageRank" - one method for link analysis) is an attempt to reflect perceived authoritativeness, over time new pages on a topic may simply reinforce that link structure by linking to the top result on Google. A newer page that is more authoritative has a problem with the headstart relative to the first result. As Google wants to deliver the most relevant page at present, a variety of factors, including a so-called "rich-get-richer" phenomenon, arise due to an implicit effect of correlation on perceived causation.

Update (fourth answer): I realized (for a comment below) that it might be useful to read Plato's Allegory of the Cave to get a sense of how to interpret correlation and causation as a result of "reflections/projections" of reality & how we (or our machines) perceive it. Correlation, strictly limited to Pearson's Correlation, is far too limited as an interpretation of the issue of misunderstanding association (broader than just correlation) and causation.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I disagree. If someone abuses the predictors to create an artificial high page rank, then the target implies the predictors, in opposite to what google intended when creating the page-rank algorithm. Since the true metric remains the same ("the right page", to which the page-rank is only an approximation), the predictors lose it's correlation and hence have to be changed. Hence google does not care about causation regarding the true metric "the right page", but to the approximate one called page-rank. $\endgroup$
    – steffen
    Commented Aug 10, 2011 at 14:29
  • $\begingroup$ No offense, but you seem a little confused about several issues. "PageRank" is a clearly defined concept and is but one predictor. The main issue you're overlooking is the industrial process involved in defining and creating a training set as well as matching user expectations. Unfortunately, the comments are a terrible place to begin a long introduction to applied machine learning. $\endgroup$
    – Iterator
    Commented Aug 10, 2011 at 15:08
  • $\begingroup$ Do you mean that in the process of predictor generation and evaluation by expert knowledge only "causation"-predictors are generated ? As long as one follows such a process and does not fall back to a trial-and-error-approach, you are correct, google cares ;). $\endgroup$
    – steffen
    Commented Aug 10, 2011 at 15:30
  • $\begingroup$ You're quite correct. The problem is that over time it gets immensely difficult to try to suss out predictors that reflect causality when there are adversaries thwarting you. If a predictor doesn't have some kind of causal explanation (because they're rarely truly causal in nature), then it is hard to justify when the baddies evolve and muck up that region of predictor space. $\endgroup$
    – Iterator
    Commented Aug 10, 2011 at 15:44
  • 2
    $\begingroup$ @Brandon: No kidding. This is best exemplified when I try looking for recent events or upcoming events. More and more frequently, I have to enter the current year or even the current MM-YYYY (or do advanced search) in order to get the relevant pages. It's a tradeoff between link structure and freshness and Google gets it wrong without more help from me. In fact, that has led me to Bing a few times, just out of irritation with trying to ignore old pages. The same is true on SO: first answers often seem to get more upvotes than later answers, which may be more correct. :) $\endgroup$
    – Iterator
    Commented Aug 10, 2011 at 19:57
5
$\begingroup$

Author of the quip here.

The comment was partially inspired by a talk by David Mease (at Google), where he said, and I paraphrase, car insurance companies don't care if being male causes more accidents, as long as it's correlated, they have to charge more. It is, in fact, impossible to change someone's gender in an experiment, so the cause could never be shown.

In the same way, Google doesn't really need to care if the color red makes someone click an ad, if it's correlated with more clicks, they can charge more for that ad.

It was also inspired by this article in Wired: The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. A quote:

"Google's founding philosophy is that we don't know why this page is better than that one: If the statistics of incoming links say it is, that's good enough."

Obviously, Google has many very smart people that know the difference between causation and correlation, but in their case, they can make plenty of money not caring about it.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ To elaborate... As I mentioned, there are a lot of people at Google who actually do care about it, as does David Mease. (Btw, he's not at Stanford, unless there's news I missed; maybe you attended his 2007 course?) But, you are right in that just as a lot of people don't know how internal combustion engines work, it doesn't affect their ability to drive. Good auto engineers & researchers, however, make it work better because they do. Same thing for the engineers & researchers at Google. Unfortunately, that Wired article is not the most articulate presentation of Norvig's thesis. $\endgroup$
    – Iterator
    Commented Aug 27, 2011 at 20:34
  • $\begingroup$ Thanks Neil for the context. I hope you don't mind me using your comment as inspiration for a question. $\endgroup$
    – Jeromy Anglim
    Commented Aug 28, 2011 at 11:22
  • $\begingroup$ @jeromy, not at all $\endgroup$
    – Neil McGuigan
    Commented Aug 28, 2011 at 19:10
1
$\begingroup$

I agree with David: The difference matters if you intend to intervene, and Google can test the results of interventions by running controlled experiments. (The optimal schedule of such experiments depends on your set of causal hypotheses, which you learn from previous experiments plus observational data, so correlations are still useful!)

There's a second reason Google might want to learn causal relationships. Causal relationships are more robust to other players' interventions. Interventions tend to be local, so they might change one part of the causal network but leave all other causal mechanisms unchanged. By contrast, predictive relationships can fail if a distant causal link is broken. The internet is constantly changing, and Google should be interested in which features of the online environment are more robust to those changes.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.