1
$\begingroup$

This is in the domain of website traffic.

Suppose I have two samples, a "pre"-survey and a "post"-survey both done before and after, respectively, a change to a website was made. Because of the nature of website traffic data, it's impossible to get the same subjects to take both surveys.

Suppose also that the sample size of the pre-survey is around 1,800 and the post-release survey's sample size is around 1,200. One question entails a 0-10 Likert scale asking how difficult or easy it was to perform a task (0 being difficult, 10 being easiest), and I would like to know whether or not there was improvement from one sample to another.

I am not familiar with working with Likert scales. But given my background (mathematical statistics is my forte), here are the concerns that come to mind:

  1. Measurement error is a really huge factor, especially given that there aren't concrete differences between individual responses on a 0-10 scale. There's not a concrete difference between, say, someone choosing a 2 over a 3. It's entirely plausible that someone could have done that depending on their mood, for example.
  2. The varying sample size is also a concern. The post-survey has a sample size that is 50% larger than the pre-survey.

What is a suitable metric for comparing these two outcomes?

Here's what my thoughts are:

  1. Percents and mean comparisons are not suitable for this, particularly due to the varying sample sizes.
  2. Traditional hypothesis testing $p$-values are not suitable for this, since given that the sample sizes are so large already, the $p$-value is going to be small anyway. (Also, as mentioned above, I don't think the mean is appropriate.)

I thought percentiles, based on the 0-10 scores, would be the most appropriate because these are not (explicitly) dependent on the sample size.

It may also be worth noting that the data are very skewed left, with around 1/3 of responses in both the pre-survey and post-survey responding with a 10.

$\endgroup$
  • $\begingroup$ so you want to compare a set of survey responses on two large independent samples? Are you doing this for research or do you need a practical presentation of the results? $\endgroup$ – JWH2006 Feb 13 '18 at 16:22
  • $\begingroup$ @JWH2006 Assume completely non-technical, non-academic audience. $\endgroup$ – Clarinetist Feb 13 '18 at 16:23
  • $\begingroup$ @JWH2006 And yes, they are essentially independent samples. The intersection between them is around 70 subjects. Given the domain in context, this is only a coincidence. $\endgroup$ – Clarinetist Feb 13 '18 at 16:25
1
$\begingroup$

I think you might consider doing a Bayesian analysis of the data with a tool like Stan. This lets you put any knowledge/assumptions of the model that generated your data into account and also lets you extract interpretable information of the form "if the model is (approximately) correct, the probability of increase of more than X points is Y% and the probability of a decrease is Z%". However, the case of a single Likert might be quite tricky to model correctly (see an aside below on how the modelling difficulties could be avoided).

A somewhat reasonable model could for example assume that there is some true continuous variable representing user's experience (let's call it $a_{1,2} \in (0,1) $ for pre- and post- condition respectively). For each person $i$, belonging to group $g(i)$, you observe a noisy realization of this (let's say $b_i$) which is Beta-distributed with mean $a_{g(i)}$ and some precision $\tau$. There is a set of thresholds $c_1 ... c_{10} \in (0,1)$ that transform $\beta$ into one of the likert answers. The quantity of interest would be $a_1 - a_2$. You could specify that with a Stan program of the form (not actually tested, but should give you a start)

data {
  int N; //Number of observations
  int<lower=0, upper=10> Y[N]; //The answers
  int<lower=1,upper=2> group[N]; //1 - pre treatment, 2 - post treatment
}

parameters {
  //Mean satisfaction per group
  real<lower=0, upper=1> a[2];

  //True satisfaction of each person
  real<lower=0, upper=1> b[N];

  //Thresholds
  positive_ordered[10] c;

  //'b' gives the same amount of information about 'a' as 'tau' coin tosses
  //with probability 'a'
  real<lower=0> tau;
}

model {
  for(i in 1:N) {
    real expected = a[group[i]];
    b[i] ~ beta(tau * expected, tau * (1 - expected));
    Y[i] ~ ordered_logistic(b[i], c);
  }

  //Priors
  //We assume both a to be more likely close to 0.5, but with a lot of leeway
  a ~ beta(2,2);
  //We assume that 'b' give roughly between (e^0) = 1 and (e^2) ~= 7.4
  //coin tosses worth of information about 'a'
  tau ~ lognormal(1,0.5);
}

You can obviously use your own assumptions about your data and as long as you can describe them in terms of probability distributions, Stan will let you make inferences with those assumptions.

After you fit such a model, you get a set of samples from the posterior distribution and thus it is straightforward to ask questions about any aspect of the model. Stan is well tested and has a good community, you should give it a try :-).

An aside: A single Likert question tends to be hard to model, so psychologist usually put multiple Likert questions representing the same construct (some coded negatively) and then take the average (after reversing the negatively-coded questions). The average tends to be a well-behaved, almost normally distributed variable. Also this reduces noise as different people often interpret the same word differently. In your case the items could have looked like:

  • The task was easy to do.
  • I did not know how to proceed with this task.
  • I felt I could complete the task confidently.
  • I did feel lost when trying to complete the task.
$\endgroup$
0
$\begingroup$

I would conduct a $\chi^2$ goodness of fit test to assess whether the distribution of Likert scores differ before and after.

In a conventional $\chi^2$ test, the test statistic is $\Sigma (observed-expected)^2/expected$. Substitute the $expected$ results with the pre-website survey percentages, and the $observed$ results with the post-website survey percentages. Now $\chi^2 = \Sigma (Post-Pre)^2/Pre$. You may have to combine some categories if any percentages in the pre-survey are zero. Remember to change the degrees of freedom accordingly.

We are testing $H_0: \chi^2=0$ (no change) vs. $H_1: \chi^2>0$ (some change in Likert distribution). Under $H_0, \chi^2 \sim \chi^2_{10}$. The degrees of freedom are $10$ as you have $11$ classes.

In terms of summarising the data, I would present the pre and post percentages graphically side-by-side. If the $\chi^2$ test is significant, then you can assess the change in the categories visually.

$\endgroup$
  • $\begingroup$ Thats likely to produce a significant results given the chi-square test's notorious sensitivity to any reasonably large sample size. $\endgroup$ – JWH2006 Feb 13 '18 at 16:22
0
$\begingroup$

I have presented technical survey data to business audiences before and here is my experience of what you might face-

  1. a lack of normal distribution: there is a reasonable chance that your data might have a bimodal distribution.

  2. The need for technical fidelity is secondary to visual representation: i.e. the fact that you used a kruskal-wallis test to compare groups rather than an ANOVA test will get an acknowledgement from a reviewer, but will mean little to a business audience. Or that you used a spearman rho over a chi-square test.

Here are my suggestions on what you should do-

  1. conduct the formal hypothesis test using a spearman rank correlation test. This is similar to chi square but works a little better with survey data (not that they will make a huge difference at your sample size). This will tell you if there is a difference in your before and after. This will likely be significant given your large sample but you will validate this in your mean testing.

  2. If there is a difference, do a mean comparison between items on your survey to find where the difference is. Is it question 1 or question 7 where your customers are happier or less happy? The fact that you have different sample sizes is not very important. You are treating them as independent. If you had a sample size of 5 and one of 20, that might be cause for concern. A sample of 1000 and 1500 might as well be the same statistically speaking (as the difference in power is miniscule).

  3. If you want, you can do subgroup analysis for your demographics. Or even something more formal like an ANOVA/K-W H test on your survey. That gives you an overall big picture difference.

  4. Graphically display the question differences using your preference of graphical representation.

If what you are presenting to is similar to annual grant reports, the funding agency does not necessarily care about your statistical sophistication.

$\endgroup$
  • $\begingroup$ JWH2006, But isn't a fundamental problem here that because the pre- subjects are almost entirely different than the post- subjects, we aren't sure how much, if any, change in mean score or in counts from pre- to post- is attributable to the intervention of changing the website? If not, how not. And if so, beyond the differences in power between an ANOVA and a non-parametric test, how much of a problem is not having the same subjects in terms of interpretability of any change if it exist? Thank you in advance $\endgroup$ – LeeZee May 17 '18 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.