I am trying to analyze data based on a sample size of around 50 and would appreciate advice on which statistical test would be most appropriate. Participants, on a single questionnaire, were asked to complete two different sections. Variable 1: rating Likert items on a scale from 1-5 (strongly agree to disagree). Variable 2: rating Likert items of a validated tool on a scale of 1-9, which is averaged to obtain a single number.

My research question is: Do those who 'agree' with the statement in the first section significantly differ in terms of their scores on the variable 2, compared to those who 'disagree'? (ex. Do participants tend to 'agree' with "I like to read books for pleasure." have higher IQs (variable 2) than those who 'disagree'?)

I was advised by a colleague to run a one-way ANOVA with variable 1 (categorical) as IV and variable 2 (continuous) as DV.

Question 1: My independent variable is the three groups: 'agree', 'neutral', and 'disagree.' (I combined strongly agree with agree because of low frequencies and likewise for strongly disagree and disagree.) However, there are many more participants in the 'agree' category than in the 'disagree' category. The distribution is, on average, around 44 agrees, 3 neutrals, and 3 disagrees. Do the (very) unequal sample sizes prevent me from performing ANOVA, or any other tests? Which test would be most appropriate for this analysis?

Question 2: I would like to run analyses on all of the Likert items, which is more than 10 statements. (ex. Seeing how IQs correlate with agreement to many statements including “I like to read books for pleasure,” “I watch a lot of TV,” “I exercise more than 30 minutes each week,” etc.) I was planning to do a Bonferroni correction to adjust for the multiple tests. Would this be correct, and help make my analyses more robust? Note: The continuous data are not normally distributed (there is negative skewness), but I seem to be able to correct the skewness using logarithmic transformation.

Any advice or suggestions for papers to read would be appreciated. Thank you!

  • 1
    $\begingroup$ ANOVA is definitely a bad choice, as your data are ordinal, not nominal. Collapsing the 5 ratings into 3 also wastes information. It's difficult to answer without information about the structure of the questionnaires. Do you assume each is unidimensional, or do you really want to treat each item as a separate construct? $\endgroup$ – Nick Stauner Jul 3 '14 at 20:07
  • 1
    $\begingroup$ @NickStauner Thanks for pointing out the ordinality of the variable. Missed that. Questionnaire: the first section is a series of 10+ Likert items. These questions were constructed by our team (not a validated tool). Each item has its own importance and we would like to treat the items separately in the analysis. The section section is the Likert scale that is averaged to obtain a single score. Does this help? $\endgroup$ – user49440 Jul 4 '14 at 12:57

A way to improve upon ANOVA with an ordinal predictor is to use dummy codes in penalized regression. Penalized regression takes advantage of the ordering among the response categories in Likert scale data: it reduces overfitting by smoothing differences in slope coefficients for dummy variables corresponding to adjacent ranks. See Gertheiss and Tutz (2009) for an overview of penalized regression for ordinal predictors. I've discussed penalized regression here a few times before:

Unfortunately, this approach probably won't do well for categories with very few observations, and I don't know of any that would. Inferential power is necessarily limited with samples so unbalanced as to have very few observations in groups of interest. Correcting for familywise error would raise the bar even further out of reach, even if that's where it belongs. Whether it is depends on whether you mean to test one big hypothesis several times on separate but related measures, or whether you want to evaluate each hypothesis test separately; familywise error adjustment isn't necessary for the latter.

If you can't collect more data, might as well give the test a shot, but give some thought to the degree of evidence you want to see. You probably won't have enough power to distinguish small differences from the null hypothesis with p < .05, so using the Neyman–Pearson framework for dichotomizing p values interpretively is probably unrealistic (more so than usual, that is). There are less polarized ways of understanding p values – one might also call them more equivocal ways, but that's probably more appropriate with relatively weak evidence anyway. For more on interpreting p values, see for instance:

The recommendation to focus on effect size estimation and confidence intervals may help here too, because it is in essence a recommendation to focus on what you can know from your data, even if it's not "enough to reject" a null hypothesis. Plotting your results may help give you a sense of what's really going on too. Don't feel that confirmatory hypothesis testing is your only option unless you have good reason to; you may be able to get some good ideas of hypotheses to test further by just exploring your data, even if you can't really conclude anything very firmly from what you have.

One last option to consider is treating your Likert scale data as continuous. This is a big assumption (that you're effectively already making with regard to your Variable 2), so keep it in mind when interpreting anything you do based on that...but it would allow you to compute correlations between each item's ratings and your Variable 2. In this case especially, you'd not want to collapse the dis/agree and strongly dis/agree categories. Also bear in mind that a t-test of a correlation assumes bivariate normality, so you might want to consider alternatives for any hypothesis tests on those effect size estimates as well.

Gertheiss, J., & Tutz, G. (2009). Penalized regression with ordinal predictors. International Statistical Review, 77(3), 345–365. Retrieved from http://epub.ub.uni-muenchen.de/2100/1/tr015.pdf.

| cite | improve this answer | |
  • $\begingroup$ Thank you! I will look into penalized regression as an alternative to ANOVA. I don't quite understand your point with not having enough power for p < .05. I was able to find statistically significant differences with ANOVA (p < .05). Were you trying to say that there isn't enough power for this finding to be meaningful? $\endgroup$ – user49440 Jul 7 '14 at 15:00
  • $\begingroup$ @user49440: No, I was just betting that your effect size would be too small to detect with N = 50. Sounds like you're on to something though! Penalized regression will just reduce the estimated differences between adjacent levels of the predictors. E.g., if the DV is lowest for people who disagreed and highest for people who agreed, penalized regression will make more moderate predictions for people who were neutral. This might make your effects seem smaller or simpler, but theoretically, that is probably the more appropriate prediction for future samples / the rest of the population. $\endgroup$ – Nick Stauner Jul 7 '14 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.