EDIT: after thinking this through a bit more, I'm convinced you don't actually need ordinal regression. See below for the reason why.
The scatterplot clearly shows a nonlinear relationship. This was to be expected, because the predictor (the X) is a continuous variable (Average.Rating) while the response is ordinal (My.Rating). A piecewise constant function is of course not linear. In this case, Spearman rank correlation would be a more adequate measure of correlation, than Pearson correlation coefficient. Anyway, you are right in your intuition that there's a connection between Average.Rating and My.Rating (the correlation test rejects the hypothesis of a null correlation coefficient), but the correlation is weak because of large variability. In other words, for a given My.Rating, the scatterplot shows clearly that there are both books with very high Average.Rating and books with very low Average.Rating. We can also see this numerically with
books <- read.csv("books.csv")
rated <- books[books$My.Rating != 0, ]
mysummary<- rated %>% group_by(My.Rating) %>% summarize(min = min(Average.Rating), IQR = IQR(Average.Rating), median = median(Average.Rating), mean = mean(Average.Rating), max = max(Average.Rating)) %>% arrange(My.Rating)
# Source: local data frame [5 x 6]
# My.Rating min IQR median mean max
# <int> <dbl> <dbl> <dbl> <dbl> <dbl>
# 1 1 3.79 0.2250 3.82 3.950000 4.24
# 2 2 3.61 0.2800 3.84 3.928095 4.45
# 3 3 3.49 0.3950 3.88 3.955686 4.36
# 4 4 3.53 0.2975 3.98 4.007604 4.49
# 5 5 3.64 0.2650 4.07 4.059800 4.44
As My.Rating increases from 1 to 5, the mean, and expecially the median of Average.Rating increase, but the dispersion of Average.Rating, quantified by either IQR or max - min, remains wide. We can see this extremely well using a boxplot:
rated$My.Rating <- as.factor(rated$My.Rating)
ggplot(rated,aes(x = Average.Rating, y = My.Rating)) + geom_boxplot()+coord_flip()
the boxplot shows clearly that, as the median of Average.Rating increases, My.Rating also increases, however there is a large dispersion among this median value. This is why you cannot get a good (accurate) regression of My.Rating over Average.Rating, whether you use ordinal regression or not. Another interesting point is that you don't have really low Average.Ratings (the minimum is 3.49), while you do have very low My.Ratings. This is in part due to the smoothing effect of the mean.
To get better results you need to add predictors, which should be correlated with your response, and possibly not too correlated among themselves. After having the possibility to look at your data, I don't think you can do much better unless you collect more data. You have a classification (the Bookshelves variable) for some books (not all), but some books belong to more than one category. You could try creating extra columns for each category, i.e., fiction, humor, etc., and for each book you would put a 1 (or a TRUE) if the book belongs to category $j$, otherwise leave a 0 (or FALSE). Then you could check if these extra variables help predicting My.Rating. This is the "simplest" improvement you can try with the data at hand. However, I think that you would have better luck if you could collect more data, because your real problem is that by aggregating all the reviews in a single number (the average rating), you throw away precious information, and you hemorrhage statistical power. Two possible roads:
- You would be able to predict My.Rating much better, if for each book, you could retrieve the ratings of all the reviewers which reviewed that book, not just the average. Let's assume you have $N$ reviewers in total. Then, for each book $i$, you should have a vector of length $N$, whose entry $r_j$ is either the review of reviewer $j$, in case she/he reviewed the book, or a
NA value in case she/he didn't. If you have access to this kind of data, you could use some actual recommender system algorithm, such as for example Collaborative Filtering. Another option would be ordinal regression, as suggested by @ArneJonasWarnke. However, I don't think you're actually interested in getting a My.Rating of exactly 5, or 4, etc. I guess you would be also happy with a rating of 4.95, i.e., with a continuous response, instead than an ordinal one. After all, if, for book $i$, the model predicts that you would give it an average rating of, say, 4.96, you would consider it worth reading, right? This means that you could simplify your life by using some kind of regularized linear regression, instead than regularized ordinal regression. In this case, LASSO and ridge regression are your friends (see for example package
glmnet). Remember that with this approach you need regularization because you will likely have much more reviewers than books, i.e., much more predictors than observations.
- a simpler (and probably less effective) alternative would be to retrieve some statistics of the review distributions, instead than just the average review. For example, if for each book you could find the number of 1 star reviews, the number of 2 star reviews, etc., then using these data as extra predictors, you could build a linear regression with some chance of success. Again, if you really wanted an ordinal response, i.e., if a response of 3.7 is unacceptable for you, then you would need to switch to ordinal regression or multiclass SVM. I really don't see why to stir up such a hornet's nest, though.