4
$\begingroup$

I have a small dataset that contains names of books, their average Goodreads ratings and the ratings I have given them. I want to know whether the average rating of a book is a good predictor of the rating I will give the same book. Here's a small sample of the data:

                                                                Title           Author
340 Quiet: The Power of Introverts in a World That Can't Stop Talking       Susan Cain
276                                                       The Witches       Roald Dahl
63                                               The 48 Laws of Power    Robert Greene
293                     Blink: The Power of Thinking Without Thinking Malcolm Gladwell
128                                                       The Martian        Andy Weir
119                                     The Design of Everyday Things Donald A. Norman
71          The Hostile Hospital (A Series of Unfortunate Events, #8)   Lemony Snicket
33                                       The Stonekeeper (Amulet, #1)    Kazu Kibuishi
369            Y: The Last Man, Vol. 1: Unmanned (Y: The Last Man #1) Brian K. Vaughan
222                                                    The Book Thief     Markus Zusak
    Average.Rating My.Rating
340           4.02         4
276           4.16         4
63            4.16         5
293           3.87         5
128           4.39         4
119           4.16         5
71            3.93         4
33            4.13         2
369           4.12         3
222           4.35         3

How do I know whether Average.Rating is a good predictor of My.Rating? I tried the cor function in R and the correlation I got was 0.1970633.

What I understand from this result is that the predictive power of a Goodreads rating is negligible. It follows that I might have a similar outcome if I had randomly picked books from a bookstore.

However, I do not feel that Goodreads ratings are as unreliable as the correlation seems to suggest. In fact, I feel that most books that have a high rating are those I have personally liked as well. Also, I intuit that selecting books based on genre and the average rating is a much better approach than randomly picking up books from a bookstore or library.

What am I missing?

Update:

Here's the scatter plot of my data: Sctterplot of My Rating vs Avg Rating

This is result that cor.test gives me:

Pearson's product-moment correlation  data:  rated$Average.Rating and rated$My.Rating

t = 2.9746, df = 219, p-value = 0.003263

alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval:
 0.06683011 0.32069438 sample estimates:
      cor 
0.1970633
$\endgroup$
  • $\begingroup$ So you don't have much data or much variation which makes this difficult. If you do a naive regression your $R^2$ is 0.08, meaning that almost all of variation in your ratings is not explained by the the average ratings. More observations is really needed though. $\endgroup$ – VCG Sep 3 '16 at 15:35
  • $\begingroup$ @VCG My dataset has 221 entries. Is that insufficient? $\endgroup$ – Green Noob Sep 3 '16 at 15:38
  • 3
    $\begingroup$ Wait - cor(X,Y) may be 0 even when Y is a deterministic function of X (thus perfectly predictable from X). Correlation is only an estimator of linear relationship. It could just be that the relationship is not linear. To have an idea , we need to see a scatterplot of average rating vs your rating. That wouldn't be my first guess, though - I think there might be two other issues with your approach. First of all, try a linear regression of your rating against two predictors - literary genre and average rating. $\endgroup$ – DeltaIV Sep 3 '16 at 15:58
  • 1
    $\begingroup$ Also, there's a reason why recommender systems are not simply logistic regressions - how are you taking into account the fact that for some of the books you like there may be very few reviews ( or no review at all), while for others there may be many reviews? $\endgroup$ – DeltaIV Sep 3 '16 at 15:58
  • 1
    $\begingroup$ @DeltaIV Thank you for your comment.I have added a scatterplot and the confidence interval to the question. I have not taken the number of reviews into account. $\endgroup$ – Green Noob Sep 4 '16 at 4:28
2
$\begingroup$

EDIT: after thinking this through a bit more, I'm convinced you don't actually need ordinal regression. See below for the reason why.

The scatterplot clearly shows a nonlinear relationship. This was to be expected, because the predictor (the X) is a continuous variable (Average.Rating) while the response is ordinal (My.Rating). A piecewise constant function is of course not linear. In this case, Spearman rank correlation would be a more adequate measure of correlation, than Pearson correlation coefficient. Anyway, you are right in your intuition that there's a connection between Average.Rating and My.Rating (the correlation test rejects the hypothesis of a null correlation coefficient), but the correlation is weak because of large variability. In other words, for a given My.Rating, the scatterplot shows clearly that there are both books with very high Average.Rating and books with very low Average.Rating. We can also see this numerically with

books <- read.csv("books.csv")
rated <- books[books$My.Rating != 0, ]
library(dplyr)
mysummary<- rated %>% group_by(My.Rating) %>% summarize(min = min(Average.Rating), IQR = IQR(Average.Rating), median = median(Average.Rating), mean = mean(Average.Rating), max = max(Average.Rating)) %>% arrange(My.Rating)
mysummary
# Source: local data frame [5 x 6]
# 
#   My.Rating   min    IQR median     mean   max
#       <int> <dbl>  <dbl>  <dbl>    <dbl> <dbl>
# 1         1  3.79 0.2250   3.82 3.950000  4.24
# 2         2  3.61 0.2800   3.84 3.928095  4.45
# 3         3  3.49 0.3950   3.88 3.955686  4.36
# 4         4  3.53 0.2975   3.98 4.007604  4.49
# 5         5  3.64 0.2650   4.07 4.059800  4.44

As My.Rating increases from 1 to 5, the mean, and expecially the median of Average.Rating increase, but the dispersion of Average.Rating, quantified by either IQR or max - min, remains wide. We can see this extremely well using a boxplot:

rated$My.Rating <- as.factor(rated$My.Rating)
ggplot(rated,aes(x = Average.Rating, y = My.Rating)) + geom_boxplot()+coord_flip()

enter image description here

the boxplot shows clearly that, as the median of Average.Rating increases, My.Rating also increases, however there is a large dispersion among this median value. This is why you cannot get a good (accurate) regression of My.Rating over Average.Rating, whether you use ordinal regression or not. Another interesting point is that you don't have really low Average.Ratings (the minimum is 3.49), while you do have very low My.Ratings. This is in part due to the smoothing effect of the mean.

To get better results you need to add predictors, which should be correlated with your response, and possibly not too correlated among themselves. After having the possibility to look at your data, I don't think you can do much better unless you collect more data. You have a classification (the Bookshelves variable) for some books (not all), but some books belong to more than one category. You could try creating extra columns for each category, i.e., fiction, humor, etc., and for each book you would put a 1 (or a TRUE) if the book belongs to category $j$, otherwise leave a 0 (or FALSE). Then you could check if these extra variables help predicting My.Rating. This is the "simplest" improvement you can try with the data at hand. However, I think that you would have better luck if you could collect more data, because your real problem is that by aggregating all the reviews in a single number (the average rating), you throw away precious information, and you hemorrhage statistical power. Two possible roads:

  1. You would be able to predict My.Rating much better, if for each book, you could retrieve the ratings of all the reviewers which reviewed that book, not just the average. Let's assume you have $N$ reviewers in total. Then, for each book $i$, you should have a vector of length $N$, whose entry $r_j$ is either the review of reviewer $j$, in case she/he reviewed the book, or a NA value in case she/he didn't. If you have access to this kind of data, you could use some actual recommender system algorithm, such as for example Collaborative Filtering. Another option would be ordinal regression, as suggested by @ArneJonasWarnke. However, I don't think you're actually interested in getting a My.Rating of exactly 5, or 4, etc. I guess you would be also happy with a rating of 4.95, i.e., with a continuous response, instead than an ordinal one. After all, if, for book $i$, the model predicts that you would give it an average rating of, say, 4.96, you would consider it worth reading, right? This means that you could simplify your life by using some kind of regularized linear regression, instead than regularized ordinal regression. In this case, LASSO and ridge regression are your friends (see for example package glmnet). Remember that with this approach you need regularization because you will likely have much more reviewers than books, i.e., much more predictors than observations.
  2. a simpler (and probably less effective) alternative would be to retrieve some statistics of the review distributions, instead than just the average review. For example, if for each book you could find the number of 1 star reviews, the number of 2 star reviews, etc., then using these data as extra predictors, you could build a linear regression with some chance of success. Again, if you really wanted an ordinal response, i.e., if a response of 3.7 is unacceptable for you, then you would need to switch to ordinal regression or multiclass SVM. I really don't see why to stir up such a hornet's nest, though.
$\endgroup$
2
$\begingroup$

Looking at your scatterplot (which is very helpful in my view), I would argue, you are not missing anything. The average rating in Goodreads is not very informative about your own rating of a book. Your sample size is with 221 observations not too small and there are books you like with high as well as low average ratings all across the board.

Of course you could do some analyses which takes into account the ordinal nature such as an ordinal regression of your own rating variable which takes only values in the set $\{1,2,3,4,5\}$ but that will not change much. Perhaps there are only few books with a high average Goodreads which you rate very badly, but I think this is a dead end.

I do not feel that Goodreads ratings are as unreliable as the correlation seems to suggest

Your feeling does not seem to be warranted by the data you presented here. Neither by the correlation you presented here, nor by the scatterplot. The graph does not suggest any nonlinear relationship as accurately suggested by DeltaIV in his comment (before the scatterplot was added to the post).

I feel that most books that have a high rating are those I have personally liked as well

If you care more about certain books than others, i.e. if you would be very sorry to not read those books, you could try to add weights, marking important books and see if both ratings give more similar results in this case? But this is of course an additional complexity.

I intuit that selecting books based on genre and the average rating is a much better approach than randomly picking up books from a bookstore or library.

This is an interesting question and we cannot answer it using the data at hand (because the books, and perhaps the ratings, in your Goodreads have not been randomly drawn/selected). If you want to be sure, try an experiment: Just randomly select and read 20 books or so (e.g. collect a large number of for example of ISBN numbers of novels and use a random generator such as the sample() function in R). Give am a rating and do not look at Goodreads before you rate the book; then look at the correlation of your rating and the Goodreads rating for those randomly selected books. Of course, come back to us and tell us whether this changed your experience regarding the Goodreads ratings! In this case, it is not primarily a statistical issue but a "problem" of non-random samples and human behavior :-)

PS: And I completely agree with DeltaIV very good comment that I would take a look if many average Goodreads ratings are based on (very) observations/ratings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.