This is in the domain of website traffic.
Suppose I have two samples, a "pre"-survey and a "post"-survey both done before and after, respectively, a change to a website was made. Because of the nature of website traffic data, it's impossible to get the same subjects to take both surveys.
Suppose also that the sample size of the pre-survey is around 1,800 and the post-release survey's sample size is around 1,200. One question entails a 0-10 Likert scale asking how difficult or easy it was to perform a task (0 being difficult, 10 being easiest), and I would like to know whether or not there was improvement from one sample to another.
I am not familiar with working with Likert scales. But given my background (mathematical statistics is my forte), here are the concerns that come to mind:
- Measurement error is a really huge factor, especially given that there aren't concrete differences between individual responses on a 0-10 scale. There's not a concrete difference between, say, someone choosing a 2 over a 3. It's entirely plausible that someone could have done that depending on their mood, for example.
- The varying sample size is also a concern. The post-survey has a sample size that is 50% larger than the pre-survey.
What is a suitable metric for comparing these two outcomes?
Here's what my thoughts are:
- Percents and mean comparisons are not suitable for this, particularly due to the varying sample sizes.
- Traditional hypothesis testing $p$-values are not suitable for this, since given that the sample sizes are so large already, the $p$-value is going to be small anyway. (Also, as mentioned above, I don't think the mean is appropriate.)
I thought percentiles, based on the 0-10 scores, would be the most appropriate because these are not (explicitly) dependent on the sample size.
It may also be worth noting that the data are very skewed left, with around 1/3 of responses in both the pre-survey and post-survey responding with a 10.