0
$\begingroup$

I have several variables that are averages of responses to a 5-star scale to rate attributes of a product. I have spent a little while reading about the potential issue of averaging a Likert scale and I believe that the validity of averageing ordinal data depends on what I want to use it for. The responses are already averaged, so I essentially have three options.

  1. Use the means as a predictor to train my model on.
  2. Don't use these variables.
  3. Do some transformation on the means, used the transformed values to train the model. I would like to train my model on the existing pricing data and then use the model to suggest prices for future products. One confounding aspect of the problem is that many of the products do not have many responses.

numresponses

This this a histogram of number of products that have a given number of responses. About 35% of the products have less than 10 responses. It worries me to use an average with few responses. If a product only had one response then the "mean" would simply be the ordinal value. On the other hand, I do think that my response ratings are reliable because I think that feature ratings of the product should not differ greatly of the ratings in my data-set. Although maybe I am not understanding the full concept of reliable in this case.

Another issue I have with this data is the leniency bias. Here is an example distribution of the ratings. This article (pg 11-2) further worries me because the leniency bias pushes my ordinal mean towards the censored region where I loose power to discriminate the latent mean.

rating dist

This distribution inspired me to look into transformations. I have thought about grouping the products by what rating they get to return to the ordinality of the groups. I am not sure how to decide the cutoffs if I did such a transformation. I also looked into the box-cox transformation. It is not able to truly normalize this data, but interestingly it does provide possible cutoff points for the distribution.

boxcox

This is the same distribution with a box-cox transform.

I got some of the background on how to use these kind of means from this post. I also found this post which is yet unanswered. There are many sources on psychometrics, but here is a paper on leniency bias. Also a post on the same.

I'd really appreciate any answer that suggests which of the three options I should go with or why. Feel free to request more info.

Edit: I am inspired by this post to transform the responses into a Poisson variable that ignores the 5-star part and counts the number of times a product is rated as "insufficient". Where insufficient is the number of times it was rated at less than 5-stars. If there are no other answers I will write accept this method as an answer.

Alternatively, the aforementioned article (pg 14) describes a very similar situation in which the authors demonstrate how a ordered-probit model preforms well. I am concerned that I will not be able to implement an ordered-probit because I do not know the ordinal variances. I would also accept an answer where someone can explain how/why I can do ordered-probit without knowledge of the variances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.