Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am currently doing a project for a class in which I am looking at yelp data. In the graph shown below, I am comparing a users average review rating with their total useful votes on their reviews. The variables in my eyes look to be definitely correlated, however the r value calculated seems to disagree.
Could this be because of the large amounts of data I have? Maybe I'm missing the point of the correlation coefficient?
A large amount of data can only help you to determine the correlation more precisely, it cannot reduce the correlation. The problem with your data seems rather to be that, yes, you have a slight positive relationship between your variables for a large number of useful votes, described by your fitted linear equation, but you also have the bulk of your data close to 0 useful votes, where the effect of this relation is small compared to the large variation in the ratings.
I would recommend the following. For visualization, use smaller dots or use a two-dimensional histogram. On the left side of the diagram you have so many superimposed circles, it's hard to see anything. Things should also get clearer if you plot the logarithm of useful votes instead of the useful votes themselves. Of course you cannot compute the log of 0, but it is possible that removing users with 0 useful votes is a good idea anyway. You can then try to make a linear fit to the relation (log useful votes) vs (rating).
Alternatively, you could rank-transform your data (replace each data point with its position in a sorted list of data points) and then attempt a linear fit. The corresponding correlation coefficient is called Spearman correlation.
hexbin package
$\endgroup$
Jun 10, 2015 at 2:59
The garden-variety Pearson correlation $r$ measures the strength of linear association between two variables $x$ and $y$. The easiest way to think of it (in my opinion) is in terms of fitting a linear model of $y$ against $x$. If the model is a perfect fit (i.e. $y$ plotted against $x$ is a straight line), $r=1$. If the fit can't get any worse (e.g. if the "true" curve is parabolic) then $r=0$. In fact, for the linear model $\mathrm{E}\left(y\right) = \beta_0 + \beta_1 x$, the familiar goodness-of-fit statistic $r^2$ is literally just the square of the correlation coefficient (hence the notation).
So two variables can be (or at least appear) very closely related, as in your case, but have near-zero correlation:
For a deeper explanation and the source of this very nice graphic, see this Wikipedia article.
It is probably because the relationship you are seeing is not linear, and the usual correlation coefficient reflects linear relationship. As @A._Donda said transform useful votesand you will see a different picture.
