How do I remove emotional bias in rating survey questions? I want to do a survey about the quality of a slot game. At the end of a game session (when player cashes out), there will be a question on screen:
"Please rate the game on a scale of 1 to 5"
Player can touch to rate. I can foresee 2 bias sources:


*

*Whether player won or lost

*Whether player encountered unusual events such as hitting a jackpot


I assume those 2 are the only major sources of bias; there may be other sources, of course.
So in addition to the rating itself, I'll also collect these data:


*

*The amount of money before and after the game session

*The number of jackpots hit


How do I construct a model that will best remove the emotional bias in rating?
Currently my approach is using multipliers. If player wins, multiply the rating score by 0.9 (to offset the positive emotion), if player loses, multiply by 1.1 (to offset the negative emotion), if player hits jackpots, multiply by 0.7. (E.g. if rating is 3/5, player wins, no jackpot hit --> adjusted rating 3*0.9 = 2.7). But a big issue with this approach is determining the multipliers.
Thank you very much!
 A: A general "rate this game" metric is hard to use, statistically. Each person may be basing that rating on different aspects of the game. I would recommend you break down that vague question into a series of very specific questions. People will tend to be less biased when they are forced to focus on something concrete. You should also ask questions that gague their mood, such as: 


*

*On a scale of 1 to 5, rate how fair you think the game is: 1=unfair to player, 3= fair/unbiased, 5=generous/biased toward player.

*Would you reccommend this to a friend? (Yes/No)


Now, if the game is indeed fair (which I am assuming it is), then the degree to which their answer to the first question deviates from 3 is an indicator of their potential bias. In addition, the answer to number 2 will address their overall tendency to rate high or low. These two questions should be correlated, with higher values to question 1 giving Yes answers to question 2 more often. 
So heres a relatively simple suggestion: Group the participants by their score to question 1. For each group, you will specify a linear "bias adjustment" function for each question. It will look like this (for the group i and question j):
$A_{ij}(R) = b_{ij}|K_i-R|$ Where $R$ is the rank given to question j by someone from group i. 
$K_i = 5$ for $i=1,2$ and $K_i = 1$ for $i=4,5$. $b_{3j}=0$ by assumption.      
Now, you will be adjusting each participant's responses to each quesiton such that $R_{ijk}^{adj}=R_{ijk}+A_{ij}(R_{ijk})$, where $R_{ijk}$ is the response from person k in group i to question j.
You will need to set the $b_{ij}$ such that the average adjusted response to question j from group i is equal to the average response from group 3 to question j. This will effectively eliminate the bias from each groups response to each question. 
Of course, there are more statistically sophisticaed and valid ways to do this, but the above is simple to explain and implement. The most important thing is to ask more focused questions and ask questions that gauge their perceptions of the fairness of the game. If you need more sophistication, I suggest you hire or ask a statistician to work with you.
