0
$\begingroup$

I am having trouble deciding the appropriate statistical test to use for a given experiment.

In the study, participants first recorded various feelings on a 4 point Likert scale.

Ex: How much do you agree with the statement "I feel calm"?

1 - Not at all

2 - Somewhat

3 - Moderately so

4 - Very much so

Then after participating in a given event, the users then recorded their responses to the questions again. I want to test whether or not the feelings before the experience are different from the feelings after the experience.

The two tests that came to mind were a Chi-Squared test and a paired t test. The Chi squared test would determine whether or not the distribution of before answers is different from the distribution of after answers. The paired t test would find the difference between each participant's before and after answer, and then test whether or not the average of these differences was equal to 0.

The detail that is throwing me off is the fact that this study included multiple participants who went through this experience multiple times. For example a given participant could have 3 observations in the data and therefore represent three observations in the data. This leads me to believe that the observations are not independent from each other as someone's difference in pre-activity and post-activity responses are most likely similar between the multiple times they participated in the activity. Would it be better to take the mean difference of a participant's pre-activity and post-activity response to use as each observation, or to keep each individual difference in the data? For example if one of the participants saw differences of 0, 2, and 1 on different days, would it be better to have this be represented by a single observation of 1 for the mean difference, or to leave 0, 2, and 1 as separate observations.

Any help in picking the appropriate statistical test to use in this situation would be greatly appreciated.

Thanks!

$\endgroup$
5
  • $\begingroup$ For the general question about testing difference btw Likert scores, see 'Related' pages in the margin and this recent Answer. // The "detail that's throwing me off" is unclear; please clarify the design of the experiment as much as you can. $\endgroup$
    – BruceET
    Apr 4, 2021 at 7:59
  • $\begingroup$ @BruceET I have tried to clarify that section of the post. Please let me know what I can clarify if you are still confused. $\endgroup$
    – Erik
    Apr 4, 2021 at 8:36
  • $\begingroup$ The "detail" has nothing to do with the Likert nature of the data but seems the more problematic thing here (as could be reflected in the title). As you suspect, dependence is an issue here, but the way how I imagine your data it will be difficult to handle as it seems that participants unsystematically may have no, one, two repetitions. Can you give us an idea about how many participants have one, two, three observations in order to know how much information is lost if just one observation is used for anyone? $\endgroup$ Apr 4, 2021 at 10:44
  • $\begingroup$ An issue with using mean differences for those who have more than one observation is to what extent a mean difference out of three difference observations or so is really of the same kind/comparable with just a single difference. Depending on the exact background and on how much information would be lost in this way,, it may be better to use only the first observation of these. $\endgroup$ Apr 4, 2021 at 10:46
  • $\begingroup$ @Lewian Yes here are the counts of number of observations per participant: 1 observation: 4, 2 observations: 5, 3 observations: 1, 4 observations: 1, 5 observations: 1, 6 observations: 1 $\endgroup$
    – Erik
    Apr 4, 2021 at 15:05

1 Answer 1

0
$\begingroup$

Making reference to the comment that gives the numbers of participants and observations, that's a small sample by any standard... and not much to go with for modelling dependence, as would be required for using more than one observation (difference) per participant. The chi-squared test is discouraged because chances are that the sample is too small for the chi-squared approximation. The paired t-test is also questionable, the t-distribution may only be very roughly correct, the nature of the data is not quantitative, and another issue is that such a "before vs. after" comparison with Likert scaled data is affected by floor and ceiling effects, i.e., somebody who chooses "very much so" before the experiment cannot increase anymore. Unfortunately I don't think there are better alternatives that work for your sample size. If you can do without a significance test, that'd be better. (There are good reasons to think that significance tests are overused and chronically misinterpreted anyway.)

The best thing you can do, in my opinion, is a good visualisation of the data, maybe a parallel coordinate plot that shown all data and how they belong together, with the possibility to show all participants for the first "before & after"-pair, and then for later pairs the plot will thin out. This also will show floor and ceiling effects clearly, and can be interpreted in an exploratory manner.

If you have to do a test by all means, I'd probably use the first "before & after" pair for each participant only (as I'd think that if a test participant has gone through the experience before, this is essentially different from doing it for the first time; so later observations as well as the mean of all of them may not be comparable with those who have just one). Then I'd do a paired t-test and interpret it "exploratorily", explaining myself that there are various reasons for doubting its validity. However, one could still think of the differences as random numbers generated from a distribution for which the t-approximation of the distribution of the test statistics may roughly work. The p-value may be so clearly significant or insignificant that it adds something to a good visualistaion, however not much should be concluded from p=0.03 or so.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.