Effect of two demographic IVs on survey answers (Likert scale)

Question

I have 4 questions in a survey with Likert-style answers (i.e. 1–5) – and the participants have also answered the demographic questions (country and study-years) – which test do I need to find if the demographics, i.e. country and study-years, have each affected answers to each of the questions significantly? Many thanks.

Answer 1

Likert ratings are ordinal data. If you want to predict your four Likert rating variables individually using your country and study-years data, you can fit four separate ordered logit or probit models. This will involve estimating separate regression coefficients for your two independent variables (IVs) in each of your four regression models. The typical significance test for regression coefficients tests whether they differ significantly from zero, i.e., whether including the given IV as a predictor improves your model of the dependent variable.

This method can't determine whether any relationships you identify are causal relationships though. Ideally, you'd need a true experimental research design to do that. Nonetheless, you might wish to assume unidirectional causality if there's really no plausible way that whatever latent factors influenced your participants' ratings could've also influenced their responses to country and study-years. For instance, there's probably no way that whatever else affects a person's Likert-rated agreement with an item like, "Tumblers are better than pumpers," could affect country of origin, but study-years probably represents a more voluntary behavior (and so would country of current residence). Therefore it might be harder to rule out causal influences from something like preference for temperate climates and country of current residence. If your variables are related in a way more like this second example, you wouldn't want to assume that country of current residence affects something like preference for temperate climates just because you can predict Likert ratings based on such a preference from country of current residence.

When using ordered logistic regression, you should probably also test the proportional odds assumption, which has been described here as follows:

One of the assumptions underlying ordered logistic (and ordered probit) regression is that the relationship between each pair of outcome groups is the same. In other words, ordered logistic regression assumes that the coefficients that describe the relationship between, say, the lowest versus all higher categories of the response variable are the same as those that describe the relationship between the next lowest category and all higher categories, etc. This is called the proportional odds assumption or the parallel regression assumption. Because the relationship between all pairs of groups is the same, there is only one set of coefficients (only one model). If this was not the case, we would need different models to describe the relationship between each pair of outcome groups. We need to test the proportional odds assumption, and there are two tests that can be used to do so...The first...[is] a likelihood ratio test. The null hypothesis is that there is no difference in the coefficients between models, so we "hope" to get a non-significant result...[The second is] a Brant test...we also "hope" that these tests are non-significant.

Answers to these related questions mention ways to use a nominal predictor like country in logistic regression, such as contrast coding (Can I use a polytomous categorical independent variable in logistic regression?) and dummy coding (Coding of categorical variables in logistic regression).

Study-years may be continuous in reality, but if you didn't collect info at least in terms of months, you might be better off treating it as ordinal. For example, the difference between "two" and "three years" could actually be anywhere from one day (or less!) to 729 days, assuming at least minimal accuracy in responses. Problems like these are especially likely if no intermediate response options were available, or if you don't intend to preserve this level of detail when coding open-ended, text-based responses (you might not want to, because people respond to open-ended questions with varying degrees of detail). Anderson (1984) refers to ordinal data such as this as a grouped continuous variable; another term for treating a continuous variable as multiple categories is polychotomization (cf. Tsuruta & Bax, 2006).

The use of ordinal, polytomous predictors in regression is tricky. Available methods include the estimation of instrumental variables, maximum likelihood estimation, and use of a conditional distribution assumption regarding the continuous underlying variable, study time (see Winship & Mare, 1984; this article also discusses the use of ordinal dependent variables). I'm not sure if any of these will work for you though, as they seem to require other observed variables you haven't mentioned.

Here's another perspective on ordinal regressors in which you might be interested:

The regression model makes no distributional assumptions about the independents, which may be discrete variables as long as other regression assumptions are met. The discreteness of ordinal variables is thus not a problem, but do ordinal variables approach intervalness? Ordinal variables must be interpreted with great care when there are known large violations of intervalness, such as where it is known that rankings obscure large gaps between, say the top three ranks and all the others. In most cases, however, methodologists simply use a rule-of-thumb that there must be a certain minimum number of classes in the ordinal independent (Achen, 1991 [reference unknown], argues for at least 5; Berry (1993: 47) states five or fewer is "clearly inappropriate"; others have insisted on 7 or more). However, it must be noted that use of 5-point Likert scales in regression is extremely common in the literature.

The point made in the last sentence here reflects the popularity of classical test theory for handling (i.e., smoothing over and often ignoring) the complexities of Likert rating data. That too is an option. Bollen and Barb (1981) demonstrated by simulation that correlating ordinal variables that are based on latent, continuous, normal distributions attenuates the true correlation more with fewer categories or stronger correlations. With five categories, attenuation is roughly 11% across original correlations ranging from .2–.9, and decreases relatively slowly with six or more. This might not be a problem for your Likert ratings if their distributions are approximately normal, but such a distribution seems unlikely for study-years.

Another option I've just found here through @Scortchi's answer to a related question (Coding for an ordered covariate) is penalized regression (Gertheiss & Tutz, 2008). Here's an excerpt from the abstract:

Based on dummy coding two types of penalization are explicitly developed; the first imposes a difference penalty, the second is a ridge type refitting procedure. A Bayesian motivation as well as alternative ways of derivation are provided. Simulation studies and real world data serve for illustration and to compare the approach to methods often seen in practice, namely linear regression on the group labels and pure dummy coding. The proposed regression techniques turn out to be highly competitive.

The process essentially smooths dummy coefficients to be less different from those for adjacent ranks, which reduces overfitting and improves predictions. It generally performs as well as or (sometimes much) better than maximum likelihood (i.e., ordinary least squares in this case) estimation of a regression model for continuous (or in their terms, metric) data when the data are actually ordinal. It appears compatible with all sorts of generalized linear models, so it should do the trick if your study-years is not actually continuous. As a bonus, this article also references a few methods for ordinal response variables like your Likert ratings, so you don't have to take my advice for that part of your problem! For even more references, see my related answer here.

References

<sup>- Anderson, J. A. (1984). Regression and ordered categorical variables. Journal of the Royal Statistical Society B, 46, 1–30.
- Berry, W. D. (1993). Understanding regression assumptions. Newbury Park, CA: Sage.
- Bollen, K. A., & Barb, K. H. (1981). Pearson's $r$ and coarsely categorized measures. American Sociological Review, 46, 232–239. Retrieved from http://www.statpt.com/correlation/bollen_barb_1981.pdf.
- Gertheiss, J., & Tutz, G. (2009). Penalized regression with ordinal predictors. International Statistical Review, 77(3), 345–365. Retrieved from http://epub.ub.uni-muenchen.de/2100/1/tr015.pdf.
- Stata data analysis examples: Ordered logistic regression. UCLA: Statistical Consulting Group. Retrieved from http://www.ats.ucla.edu/stat/sas/notes2/.
- Tsuruta, H., & Bax, L. (2006). Polychotomization of continuous variables in regression models based on the overall C index. BMC Medical Informatics and Decision Making, 6(41). Retrieved from http://www.biomedcentral.com/1472-6947/6/41/.
- Winship, C., & Mare, R. D. (1984). Regression models with ordinal variables. American Sociological Review, 49, 512–525. Retrieved from http://scholar.harvard.edu/files/cwinship/files/asr_1984.pdf.</sup>