I'm composing a survey where I compare two mobile apps and am not sure how to formulate the questions.
Should I compare the apps directly like this: "Which app was more intuitive to use?" 1 (App A) ... 5 (App B)
Or pose one question per app like this: "How do you rate the intuitiveness of App A/B?" 1.. (poorly) .. 5(very well)
Which is more effective to evaluate statistically? I developed one of the apps so my hypothesis is that the ratings on my app are equal or better than existing ones.
The best approach depends on two things: (a) What you want to know and (b) what kinds of questions the interviewees are willing and able to answer usefully. Also, you are correct to think ahead how you will analyze the data from the survey once you get the results.
"Which app was more intuitive to use?" 1 (App A) ... 5 (App B)
This tends to ask the subject to make a choice. Presumably, A subject who can see no real difference between A and B as to 'intuitive use' will choose 2. (You'd have no way to know whether a subject really dislikes both A and B, perhaps A a little less than B, or whether a subject really likes both, perhaps with a slight edge for A.)
You could do a one-sample Wilcoxon test of the null hypothesis that the median score is 2.
For example, if you got 100 scores with the tally below, then there is very strong evidence that subjects prefer A (results from R):
1 2 3 4 5
12 14 19 32 23
Wilcoxon signed rank test with continuity correction
data: x
V = 2224, p-value = 0.006126
alternative hypothesis: true location is not equal to 3
"How do you rate the intuitiveness of App A/B?" Perhaps you would ask each subject to rate A and B on two separate 5-point Likert scales: -2=Extremely unintuitive, ..., +2=Extremely intuitive.
A subject who finds no real difference as to 'intuitive use' could give equal ratings to both. A subject who finds a difference has a way to show how big the difference is. Perhaps tangentially, subjects who like or dislike both have a way to express that.
You could summarize the data as a pair of Likert scores for each subject. Then you could look as differences: Score A - Score B, which could be -4, -3, ..., + 4. This could be analyzed using a one-sample Wilcoxon test. The null hypothesis would be that the median score is 0.
Perhaps tangentially, you could sum the scores for each subject getting sums -4 through +4, an overall favorability score for the two tests. This could also be analyzed with a one-sample Wilcoxon test.
For example, suppose tallies of 100 responses for differences D = A - B were as below:
-4 -3 -2 -1 0 1 2 3 4
3 2 3 16 17 24 18 12 5
Then a Wilcoxon signed rank test shows strong evidence favoring A.
wilcox.test(d)
Wilcoxon signed rank test with continuity correction
data: d
V = 2629.5, p-value = 4.309e-05
alternative hypothesis: true location is not equal to 0
Also, if tallies of sums are as shown below, then there may be an overall positive impression of the two apps. (Perhaps subjects just prefer to 'be nice', giving generally positive ratings, or perhaps they would happily purchase either app. If the main purpose of the survey is to know whether subjects would buy the apps when available, you should ask that as a separate question.)
-3 -2 -1 0 1 2 3 4
5 3 14 20 22 21 13 2
Wilcoxon signed rank test with continuity correction
data: s
V = 2488, p-value = 2.353e-05
alternative hypothesis: true location is not equal to 0
Use the second option. You will have a list of "intuitiveness values" for app A, and a list of values for app B. You can then use a t-test to evaluate if the difference in the mean "intuitiveness values" of each app is statistically significant. You will of course need to ensure you've satisfied the assumptions of a t-test which are a normal distribution in your data, and random sampling.
I'm not sure what statistical test you could use for the first option. I think the formulation of the first style question will complicate things.
TL;DR: If your goal is to determine which app is more intuitive, a question asking for a direct comparison is sufficient. You can fit a Bernoulli distribution to your data via maximum likelihood for a frequentist confidence interval around the proportion of users who find an app as or more intuitive than another (a Bayesian alternative is provided below).
The long version:
If your goal is simply to determine which app is more intuitive, asking for a direct comparison, such as the following question, would be sufficient:
Which application did you find more intuitive to use?
- Application A was more intuitive.
- Application B was more intuitive.
- Application A and B were equally intuitive.
This question asks for a direct comparison so you can infer if users find A is more intuitive than B; however it doesn't tell you the magnitude of the difference, nor does it tell you whether or not either are intuitive in the first place (e.g., A and B can be equally intuitive in that they are both not intuitive).
I am not aware of any methods to meaningfully measure the magnitude of intuitiveness (though some may exist in the psychological literature), but if you need to know if users think the apps are intuitive PERIOD you might consider prefacing the direct comparison with a pair of yes/no or Likert scale questions. For example, to ask about application A:
Did you find application A intuitive to use?
- Yes
- No
Or
Please rate your agreement with the statement "I found application A intuitive to use"
- Strongly Agree
- Agree
- Neither Agree nor Disagree
- Disagree
- Strongly Disagree
The second question appears like it may provide a sense of which application is more or less intuitive and by how much; however a challenge to address is whether or not you believe your survey respondents will be able to consistently measure the "intuitiveness" of the two applications. In other words, will their evaluation of the intuitiveness of application A be based on the same criteria as application B. This is true both between questions and across survey respondents (i.e. a respondent needs to be self-consistent, and all respondents should mean roughly the same thing by their stated response). If you are not confident that responses to the two questions will be commensurable, then a question asking for a direct comparison is likely safer (in the sense of producing a meaningful result).
Edit: With regards to a statistical test for the direct comparison, fitting a Bernoulli distribution to your data (assuming responses are IID) via maximum likelihood can give you a confidence interval on the probability (or proportion) of respondents who will find app A (or B) equally or more intuitive. To facilitate all three responses (A is more intuitive than B, B is more intuitive A, A and B are equally intuitive) you could use a categorical distribution.
The problem you have framed can easily adopt a Bayesian approach by using a beta distribution as a prior for the Bernoulli distribution where the alpha parameter is equal to the count of responses for "A is more intuitive than B" and the beta parameter is equal to sum of the count of the response for "B is more intuitive than A" and "A and B are equally intuitive". Obviously you can swap A and B depending on which app you need to determine is at least or more intuitive. You can extend the Bayesian approach to all three responses via a Dirichlet distribution.
