4
$\begingroup$

I've got product ratings for a few thousand products. The number of ratings for each product varies from zero to about fifty. I want to find the expected value of product rating for each product. If there are lots of ratings for the product I'd expect the expected value to be the average of the ratings for the product, but if there are only a few I'd expect the expected value to be closer to the average of all ratings. How do I calculate the true expected value? Please be gentle: I'm no statistician or mathematician.

Edit 1: Joris's answer below maintains I can't calculate expected value because by definition that means I must have the entire population. In that case please can you tell me how to calculate the quantity that is similar to expected value in spirit, does not require the entire population, and can make use of prior information.

Edit 2: I would expect that if each product's ratings have low variance ratings, or if there is a very high variance between different products' ratings, then the measured ratings are more significant.

Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
4
$\begingroup$

Incorporating a prior is one way to 'make up' for small samples. Another is to use a mixed model, with an intercept for the mean structure and a random intercept for each product. The estimate of the population mean plus the predicted random effect (BLUP) then offers a form of shrinkage, where values for products with less information are shrunk more toward the overall sample mean than those based on more information. This method is common in, for example, Small Area Estimation in survey sampling.

Edit: The R code might look like:

library(nlme)
f <- lme(score ~ 1, data = yourData, random = ~1|product)
p <- predict(f)

If you go this route the assumptions are:

  • independent, normal errors with expected value 0 and constant variance for all observations
  • normal random effects with expected value 0

Violations of these can generally be modeled, but of course with that comes added complexity...

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. I've looked at some of your suggestions and they seem to be along the lines I was thinking. But can you recommend anything more specific and suitable for the lay person that I am. $\endgroup$
    – bart
    Sep 1, 2010 at 18:56
  • 1
    $\begingroup$ @bart -- I wish I could offer a safe automated way to do this but I don't think it exits. See above for edits w/ a little more detail. $\endgroup$
    – Kingsford Jones
    Sep 1, 2010 at 22:54
  • $\begingroup$ You forget that some products don't have a score, or only very few. Lme is very likely to get convergence problems, and the estimates of the standard errors cannot be trusted. $\endgroup$
    – Joris Meys
    Sep 2, 2010 at 11:00
  • $\begingroup$ @Joris-Without a prior you can't include products with 0 information, but because there are thousands of products, having few observations per product is not a problem (even many with 1 obs will be OK). But still, without being familiar with the data, the context of the problem, and desired output/inferences we are only speculating as to what is appropriate. $\endgroup$
    – Kingsford Jones
    Sep 2, 2010 at 15:44
3
$\begingroup$

The "true" expected value cannot be calculated. You can estimate it using the mean of the ratings for each product, and get an idea about the position by calculating the 95% confidence interval (CI) on the mean.

This is done by

$CI \approx avg \pm 2 * \frac{SD}{\sqrt{n}}$

with n being the number of ratings, SD the standard deviation and avg the average. More correct would be to use the T-distribution, where you use the 2.5% and 97.5% quantile of the T-distribution with degrees of freedom equal to number of observations minus one.

$CI = avg \pm T_{(p=0.975,df=n-1)} * \frac{SD}{\sqrt{n}}$

For 10 ratings, $T_{(p=0.975,df=n-1)}$ is 2.26. For 50 ratings, it is 2.01.

There's a chance of 95% this confidence interval contains the true value. Or, to please Nèstor: if you do this experiment 10,000 times, 95% of the confidence intervals you construct this way will contain the true value for the expected value.

You assume here that the distribution of the average is normal. If you have a very low amount of ratings, the SD can be estimated wrongly.

In that case, you could estimate an "overall" standard deviation on the scoring, and use that to calculate the CI. But keep in mind that this way you assume that the standard deviation is the same for every product.

In extremis, you could resort to bootstrapping to calculate the CI for every product. This will increase the calculation time substantially, and won't be adding any value for products with enough ratings.

Improve this answer
$\endgroup$
15
  • $\begingroup$ As a newbie student maybe I shouldn't be too picky, but I'm not very happy with this answer. Why can't an ev be calculated? Can you prove that statement? Also 95% CI seems plain arbitrary. $\endgroup$
    – bart
    Sep 1, 2010 at 12:57
  • $\begingroup$ E[X] is a property of the pop that you are drawing inferences about. x-bar is an unbiased estimator of that value. You don't have to use 95% -- adjust t accordingly. But note the SE of x-bar is s/sqrt(n) not s/n. I'll see if I can edit this. $\endgroup$
    – Kingsford Jones
    Sep 1, 2010 at 15:31
  • $\begingroup$ @Bart : an expected value is about the population, and a theoretical value. It can be seen as the limit of the sample mean when the sample size goes to infinity. You need to estimate it using the mean, but you can't calculate it unless you know the complete population. Which you don't. See : en.wikipedia.org/wiki/Expected_value $\endgroup$
    – Joris Meys
    Sep 1, 2010 at 15:40
  • $\begingroup$ @Joris The wiki entry is much too advanced for me. Maybe I'm not asking for the right quantity. If by definition you need the entire population to calculate ev, I'd like a similar quantity that allows me to incorporate priors when my data is incomplete. So I can update the question, what would I call that? $\endgroup$
    – bart
    Sep 1, 2010 at 16:22
  • 2
    $\begingroup$ @bart, think of this question: what is the expected value of the height of an American? The only way to know this value exactly would be to ask every single person his height and take the average. That's the expected value---it's a property of the full population. That's really hard. Instead, we get a sample, a subset of our population, ask them their heights and take the average. This is the sample mean---a property of our sample and an estimate of the expected value. As our sample gets larger, we should get a sample mean that gets closer to the true expected value (the law of large numbers). $\endgroup$
    – Charlie
    Sep 1, 2010 at 19:03
0
$\begingroup$

I haven't looked into it much, but this article on Bayesian rating systems looks interesting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the link. I agree that the number of ratings for the product compared to average number for all products is significant. But the variance must be significant too. Suppose every product had zero variance in its ratings, then a single product rating would be sufficient to provide the ev. $\endgroup$
    – bart
    Sep 1, 2010 at 12:50
  • $\begingroup$ Good point. I just wanted to flag that the Bayesian option seems like the way to go. Of course @Kingsford has now provided a more rigorous explanation. $\endgroup$
    – Jeromy Anglim
    Sep 2, 2010 at 8:49
0
$\begingroup$

Ha! I've answered my own question. Simon Funk figured this out for the Netflix challenge here. See the paragraph commencing "However, even this isn't quite as simple as it appears". But I'm having difficulty proving it algebraically: maybe you guys would like to take that on.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ You should delete this 'answer' and add it to your question as you have not really found the answer as evident by the last line. $\endgroup$
    – user28
    Sep 2, 2010 at 17:48
  • $\begingroup$ Nice find, but when Simon writes "view that single observation as a draw from a true probability distribution who's average you want...and you can view that true average itself as having been drawn from a probability distribution of averages" he is describing a mixed or multilevel model. I don't think there's a need to guess that "K=25 seems to work well" because the best linear unbiased predictor equations were worked out more than 60 years ago (BLUP). Excellent Bayesian estimators exist as well. As Brad Efron said, "Those who ignore Statistics are condemned to reinvent it" $\endgroup$
    – Kingsford Jones
    Sep 2, 2010 at 18:00
  • $\begingroup$ @Kingsford What's worong with K = Vb/Va? $\endgroup$
    – bart
    Sep 2, 2010 at 18:57
  • $\begingroup$ @Srikant It's the best "answer" I've got so far. I think I'll be trying it out because: 1. As currently stated Jorly's doesn't actually provide the ev, and there is no indication of how prior information is utilised. 2. Kingsford's answer could suffer convergence problems apparently, and there is a lot of work for me to figure out what it all means. 3. Simon's solution worked well for him, and I trust Simon. 4. I think in time I'll get a proof of his method 5. The method uses all the available data and is correct at the extrema when individual variance is small and large $\endgroup$
    – bart
    Sep 2, 2010 at 19:28
  • $\begingroup$ @bart - I don't think you'll run into convergence problems, but you're right that it takes awhile to learn the methods. As for what's wrong with Vb/Va, I'm not sure how to answer because it's not clear to me how those values are being calculated. But under the nested Gaussian assumption described, one good way to define 'best' is in terms of minimizing mean squared error (MSE). This is what the BLUP equations do. See, for example, here. $\endgroup$
    – Kingsford Jones
    Sep 2, 2010 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.