I have a survey with a list of questions in which respondents can pick their perceived personality traits. Then I have a question that asks respondents about their favorite in-game character's perceived personality and a question that asks respondents about their least favorite in-game character's perceived personality trait. My hypothesis is: People will prefer characters that they perceive to be similar to themselves, and dislike ones that they perceive to be different. What is the best way that I can conduct this statistic test?

Example: One respondent may answer like this:

  • Extrovert (1-7): 2
  • Dependable (1-7): 5
  • Reserved (1-7): 4

Favorite Character:

  • Extrovert: no
  • Dependable: yes
  • Reserved: no

Least Favorite Character:

  • Extrovert: no
  • Dependable: no
  • reserved: no

I know how to test simple hypotheses about two simple numeric variables, but this looks a lot more complex.


There are many ways to approach this problem. One simple and flexible approach is as follows:

1) Come up with a way of computing a coefficient c for each user, such that this coefficient should usually be bigger if your hypothesis is true than if not. For example, you could use

c = sum_{feature in {Extrov, Dep, Res}} (how much likes feature)(favorite character has feature - least favorite character has feature).

Designing this coefficient is up to you. The important thing is that you expect this coefficient to be big when your hypothesis is true.

2) Then compute the average of this coefficient over your data.

3) The question you now need to answer is: Is the coefficient you got above unusually big? You can do this with a permutation test. Specifically, permute all the answers to the three questions (i.e., for each question individually, reassign all the answers to this question to new people) and then re-compute your coefficient. Do this repeatedly. If the coefficient you got in part (2) is bigger than 97.5 of the coefficients you get by permuting your data, then you may be able to report a significant effect.

  • $\begingroup$ Thank you that is great, is there any way to estimate practical significance with this approach? $\endgroup$ – user10165 Apr 25 '13 at 18:16
  • $\begingroup$ Practical significance is always somewhat subjective... It seems that the best that you can do here is to compare the value of the coefficient c for your actual data with the mean value of c over permuted data. Depending on how you designed c, this may give you a useful notion of practical significance. $\endgroup$ – Stefan Wager Apr 25 '13 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.