6
$\begingroup$

I'm analysing survey results with most responses coming in the form of Likert scales. Many of these scales have either very few or 0 responses in the bottom categories. As you can imagine, this is leading to some complications when I try to run my models. I’m still getting encouraging results, but want to be sure that my information is valid before reporting it. Below is a scenario I’ve been working with over the last couple of days:

I’m using SPSS to run an ordinal regression with two predictors. In this case, the predictors themselves are actually responses on a Likert scale (but entered into the model as nominal variables). My DV is, of course, also an ordinal scale. My two predictors each have five categories (levels on the scale). My dependent variable also has 5 levels. Just like with my predictor variables, the dependent variable has very few observations in the bottom categories. In fact when I run the regression, it says that 47.5% of cells have frequencies of 0. Yet all my coefficients are significant, the overall model fit (-2 log likelihood) is significant at .000, and the odds ratios (exponentiated form of my coefficients) all seem reasonable. The model looks like a good one other than these cells with frequencies of 0.

My test is failing the proportional odds assumption, which says that the coefficients for each predictor category must be equal across all DV levels. I know this based upon the results of the Test of Parallel lines, which SPSS reports as part of the ordinal regression output. So, on the recommendation of an article I found online, I've done two things to explore further. First, I've run separate logistic regressions with new dependent variables, each one representing a cutpoint in my original DV— in other words, they indicate whether Y is less than each of my original DV categories (excluding the bottom one). So my new DV's are level 2 or above vs. not; level 3 or above vs. not; etc.. These did not yield significant relationships for most IV-DV combinations (cells). The idea is to compare the odds ratios across the different cutpoints to see if they’re fairly constant. In my case, since few are significant, they’re not.

The second thing I’ve done is estimated separate ordinal regressions using my original dependent variable— I did one model for each category in my predictors, coded as dummies. So, in 10 separate models (2 predictor variables with 5 categories each), my single predictor would be: a 1 for level 2 and a 0 for all other levels; or a 1 for level 3 and a 0 for all other levels; etc.. For most of these categories, the parallel lines is failed (i.e. the null that the proportional odds assumption is upheld is proven true – a good thing). However, in a couple of these categories, I have no observations (nobody responded Very Poor or Poor on one of my predictor Likert scales). Thus I cannot get a parallel lines P-value for these categories.

My question has two parts.

One is whether it is the bottom levels of the predictor variables that are causing the parallel lines test to fail… and if the reason is that there are no observations in these categories, whether I can still use the overall odd's ratios from my full model. I think this should be no problem since these categories automatically drop out of the model.

The second question is whether instead, it might be the low/0 frequencies in the bottom levels of my DV that is causing the parallel lines test to fail. I don’t think it is based on the fact that the test is passed for all predictor variable categories that have observations in them. I have tried combining the bottom categories of my DV, and this decreases the % of cells with frequencies of 0, but does not totally eliminate the problem.

Many thanks for taking the time to consider my question. I would be tremendously grateful for any guidance that you can provide.

$\endgroup$
3
$\begingroup$

First off, are your two independent variables being adjusted as factors or numerically coded responses and is there an interaction term for the two? The reason I ask is because the test of proportional odds grows very sensitive with small cell counts. For this reason, I often find it justifiable to adjust input variables as their ordinally coded values (1: poor, 2: fair-to-poor, etc.). Doing so allows information to be borrowed across groups, proportionality is assessed so that an associated difference in the odds of a more favorable response comparing units differing by 1 in the predictor are consistent with odds of an even more favorable response (the rough and contrived interpretation of the test of proportional odds).

If your numeric coding still fails to give valid proportionality, it is possible to get consistent cumulative odds ratios estimates by collapsing adjacent categories like the two bottom box responses.

Thirdly, another powered test of association between an ordinal response and two ordinal factors is a plain old linear regression model. Using robust standard errors, you get valid confidence intervals despite the distribution of the errors. This tends to be less powerful that categorical methods, but with fewer pitfalls due to zero cell counts.

Lastly, as a comment, robust standard errors allow consistent estimation of the mean model in most circumstances. I'm not sure if these are implemented in SPSS, but R and SAS use these frequently. As with the proportional hazards assumption in the Cox model, when this "model based assumption check" fails, it does not mean the model results are entirely invalid, it's just that the effect estimates are "averaged" over their inconsistent proportionality. For instance, if proportional odds model has excessive numbers of respondents giving top box responses, and a predictor shows a large association for the top box response but smaller association for other cumulative measures, then you'll find that the cumulative odds ratio is a weighted combination of the several thresholded odds ratios, with a higher weight placed upon the top box OR.

$\endgroup$
  • $\begingroup$ Thank you for the insight! You've been very helpful. My independent variables are in fact coded numerically. There are no interaction terms in my model. Would this complicate information being "borrowed across groups" and be detrimental? I will collapse the bottom categories as a final step if necessary. But what I'm really curious about is your last paragraph. I've chosen the complementary log-log option for my regression... is this what you're referring to when you talk about a greater weight being putting on the top box OR? If this is the case, what does that mean? Can I be confident that m $\endgroup$ – Jeff Nov 13 '13 at 2:57
  • $\begingroup$ In converting Jeff's post to a comment, it got truncated. Here is the rest: Can I be confident that my issue of low frequencies in the bottom categories is being accounted for? The other thing that's bothering me is that, when I ran separate logistic regressions on each of the cumulative cut-points in my DV, almost none of my IV categories showed significant associations. But in the overall ordinal regression model they do. Any idea what could be the reason for that? $\endgroup$ – cardinal Nov 13 '13 at 3:13
  • $\begingroup$ Hmm... it's curious you use the cloglog link b/c that usually models binary outcomes. I think proportional odds requires that you use the cumulative logit, at least to conserve interpretation of the model parameters. This might also explain the overall significance that's not found in any thresholded logit models, although theoretically what you describe is perfectly plausible. As far as continuously coding factors to borrow information across groups, if it makes scientific sense then do so. Technically it's very similar to how the Prop Odds model handles the outcome. $\endgroup$ – AdamO Nov 13 '13 at 18:36
  • $\begingroup$ Great, thanks again! C-log-log is actually part of the ordinal regression package in SPSS... you can choose this link function along with several others. So I imagine it must be compatible. To your second point, I've coded my factors numerically but entered them into the model as nominal variables. I suppose there's no theoretical reason why I couldn't treat them as continuous. Yet they only have 5 levels, with the bottom two having very few observations, so I though it better to make them categorical. It sounds like only continuous factors allow for the "borrowing" you speak of, no? $\endgroup$ – Jeff Nov 13 '13 at 22:41
  • $\begingroup$ In my experience, cloglogs are not well suited to this problem, they're used for discrete survival problems. Furthermore, they don't have any analogue to logistic regression so you can't compare your logit models to cloglog models. In the context of a cumulative link model, I have no idea how to interpret the model coefficients. Indeed only continuous factors allow borrowing btn groups. It greatly improves the effective degrees of freedom, as well, which leads to stable estimation of cumulative odds ratios. $\endgroup$ – AdamO Nov 13 '13 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.