5
$\begingroup$

I am currently doing a project for a class in which I am looking at yelp data. In the graph shown below, I am comparing a users average review rating with their total useful votes on their reviews. The variables in my eyes look to be definitely correlated, however the r value calculated seems to disagree.

graph

Could this be because of the large amounts of data I have? Maybe I'm missing the point of the correlation coefficient?

$\endgroup$
2
  • 2
    $\begingroup$ Your question seems somehow related to this one: stats.stackexchange.com/questions/32063/… It seems there is a subset of correlated points. $\endgroup$
    – Simone
    Jun 10, 2015 at 2:18
  • 2
    $\begingroup$ Visually the relationship looks very weak on the right-hand side and if the regression line had been horizontal (something like $y=4$) I would not have been surprised $\endgroup$
    – Henry
    Jun 10, 2015 at 6:32

3 Answers 3

5
$\begingroup$

A large amount of data can only help you to determine the correlation more precisely, it cannot reduce the correlation. The problem with your data seems rather to be that, yes, you have a slight positive relationship between your variables for a large number of useful votes, described by your fitted linear equation, but you also have the bulk of your data close to 0 useful votes, where the effect of this relation is small compared to the large variation in the ratings.

I would recommend the following. For visualization, use smaller dots or use a two-dimensional histogram. On the left side of the diagram you have so many superimposed circles, it's hard to see anything. Things should also get clearer if you plot the logarithm of useful votes instead of the useful votes themselves. Of course you cannot compute the log of 0, but it is possible that removing users with 0 useful votes is a good idea anyway. You can then try to make a linear fit to the relation (log useful votes) vs (rating).

Alternatively, you could rank-transform your data (replace each data point with its position in a sorted list of data points) and then attempt a linear fit. The corresponding correlation coefficient is called Spearman correlation.

$\endgroup$
2
6
$\begingroup$

related ≠ correlated

The garden-variety Pearson correlation $r$ measures the strength of linear association between two variables $x$ and $y$. The easiest way to think of it (in my opinion) is in terms of fitting a linear model of $y$ against $x$. If the model is a perfect fit (i.e. $y$ plotted against $x$ is a straight line), $r=1$. If the fit can't get any worse (e.g. if the "true" curve is parabolic) then $r=0$. In fact, for the linear model $\mathrm{E}\left(y\right) = \beta_0 + \beta_1 x$, the familiar goodness-of-fit statistic $r^2$ is literally just the square of the correlation coefficient (hence the notation).

So two variables can be (or at least appear) very closely related, as in your case, but have near-zero correlation:

enter image description here

For a deeper explanation and the source of this very nice graphic, see this Wikipedia article.

$\endgroup$
2
$\begingroup$

It is probably because the relationship you are seeing is not linear, and the usual correlation coefficient reflects linear relationship. As @A._Donda said transform useful votesand you will see a different picture.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.