The best approach depends on two things: (a) What you want to know and (b) what kinds of questions the interviewees are willing and able to answer usefully. Also, you are correct to think ahead how you will analyze the data from the survey once you get the results. **"Which app was more intuitive to use?" 1 (App A) ... 5 (App B)** This tends to ask the subject to make a choice. Presumably, A subject who can see no real difference between A and B as to 'intuitive use' will choose 2. (You'd have no way to know whether a subject really dislikes both A and B, perhaps A a little less than B, or whether a subject really likes both, perhaps with a slight edge for A.) You could do a one-sample Wilcoxon test of the null hypothesis that the median score is 2. For example, if you got 100 scores with the tally below, then there is very strong evidence that subjects prefer A (results from R): 1 2 3 4 5 12 14 19 32 23 Wilcoxon signed rank test with continuity correction data: x V = 2224, p-value = 0.006126 alternative hypothesis: true location is not equal to 3 **"How do you rate the intuitiveness of App A/B?"** Perhaps you would ask each subject to rate A on a 5-point Likert scale: -2=Extremely unintuitive, ..., +2=Extremely intuitive. A subject who finds no real difference as to 'intuitive use' could give equal ratings to both. A subject who finds a difference has a way to show how big the difference is. Perhaps tangentially, subjects who like or dislike both have a way to express that. You could summarize the data as a pair of Likert scores for each subject. Then you could look as differences: Score A - Score B, which could be -4, -3, ..., + 4. This could be analyzed using a one-sample Wilcoxon test. The null hypothesis would be that the median score is 0. Perhaps tangentially, you could sum the scores for each subject getting sums -4 through +4, an overall favorability score for the two tests. This could also be analyzed with a one-sample Wilcoxon test. For example, suppose tallies of 100 responses for differences D = A - B were as below: -4 -3 -2 -1 0 1 2 3 4 3 2 3 16 17 24 18 12 5 Then a Wilcoxon signed rank test shows strong evidence favoring A. wilcox.test(d) Wilcoxon signed rank test with continuity correction data: d V = 2629.5, p-value = 4.309e-05 alternative hypothesis: true location is not equal to 0 Also, if tallies of sums are as shown below, then there may be an overall positive impression of the two apps. (Perhaps subjects just prefer to 'be nice', giving generally positive ratings, or perhaps they would happily purchase either app. If the main purpose of the survey is to know whether subjects would buy the apps when available, you should ask that as a separate question.) -3 -2 -1 0 1 2 3 4 5 3 14 20 22 21 13 2 Wilcoxon signed rank test with continuity correction data: s V = 2488, p-value = 2.353e-05 alternative hypothesis: true location is not equal to 0