Statistical test for preference data through average differences or is other method more suitable? I need your help. I'm designing a questionary and there I want test which version the people prefere. Which of the following methods is better and which statistical test for significance in each method is most suitable?
Method 1: Rating of each version and calculation if the averages of each version are significantly different.


*

*How do you like the following versions? 1-5 scale with 1: poor and 5: good, each version is rated


*

*version 1

*version 2

*version 3



So in the end each version will have a rating average and then I can calculate if the averages are significantly different but with which statistical test and is method 1 even a good choice?
Method 2: 


*

*Which version do you prefere the most? single choice, so one can choose only single version of three


*

*version 1

*version 2

*version 3



So in the end I will have votings for example: 10 for version 1, 20 for version 2 and 30 for version 3 (total 60 votings). Which statistical test is suitable to tell if version 3 is significantly more desirable than version 1 and 2 and is method 2 a better choice than method 1?
 A: 1) Is method 1 or method 2 better? In my opinion this depends on the background and what you want. Method 1 gives more differentiated information, method 2 focuses on the decision problem and forces people to not sit on the fence (by giving 2 or 3 versions the same score). This is a subject matter/psychological issue, not a statistical one.
2) In both cases there are standard tests (see below) for either testing the null hypothesis whether any two specified versions are rated the same, or whether all three of them are the same. The standard tests do not test the null hypothesis that the highest rated/most chosen version is equal to the second best, because this involves data-dependent selection of the null hypothesis. If you're fine with testing the H0 that all three are equal, so be it. In many applications that's enough. If you want to test the highest against the second highest, one option is to use a test that compares two versions with Bonferroni correction, i.e., at level $\alpha/3$ if in fact you want to test at effective level $\alpha$ (because you pick one of three possible comparisons). Chances are something better can be done, but it would be non-standard and maybe complicated.
3) Data from method 1 could be analysed using repeated measures ANOVA (testing "all three equal") or a matched two-sample t-test (testing equality of two of the versions). Some people will say that this is not appropriate because of ordinal data. I'm not worried much because if you have a sample size that isn't very low, the test statistic for a 5-point Likert scale as you seem to use will typically be distributed almost identically to that one for normally distributed data as assumed by the test (Central Limit Theorem and all that). The Friedman test transforms data to ranks and does a similar thing but is supposedly admissible for ordinal data. As your scores will often be equal and ranks need to be averaged (which a software will do automatically) I doubt that this test will perform better in terms of power and level than the ANOVA here, but some people will prefer it. (The two-sample version is Wilcoxon signed rank test.)
4) Data from method 2 could be tested by a $\chi^2$ goodness of fit-test of the null hypothesis that all three version choice probabilities are equal 1/3 (chisq.test; can also be used with two proportions).  
