Continuous dependent variable with ordinal independent variable

Question

Given a continuous dependent variable y and independent variables including an ordinal variable X₁, how do I fit a linear model in R? Are there papers about this type of model?

Answer 1

@Scortchi's got you covered with this answer on Coding for an ordered covariate. I've repeated the recommendation on my answer to Effect of two demographic IVs on survey answers (Likert scale). Specifically, the recommendation is to use Gertheiss' ⁽²⁰¹³⁾ ordPens package, and to refer to Gertheiss and Tutz ^(2009a) for theoretical background and a simulation study.

The specific function you probably want is ordSmooth^*. This essentially smooths dummy coefficients across levels of ordinal variables to be less different from those for adjacent ranks, which reduces overfitting and improves predictions. It generally performs as well as or (sometimes much) better than maximum likelihood (i.e., ordinary least squares in this case) estimation of a regression model for continuous (or in their terms, metric) data when the data are actually ordinal. It appears compatible with all sorts of generalized linear models, and allows you to enter nominal and continuous predictors as separate matrices.

Several additional references from Gertheiss, Tutz, and colleagues are available and listed below. Some of these may contain alternatives – even Gertheiss and Tutz ^(2009a) discuss ridge reroughing as another alternative. I haven't dug through it all yet myself, but suffice it to say this solves @Erik's problem of too little literature on ordinal predictors!

References

<sup>- Gertheiss, J. (2013, June 14). ordPens: Selection and/or smoothing of ordinal predictors, version 0.2-1. Retrieved from http://cran.r-project.org/web/packages/ordPens/ordPens.pdf.
- Gertheiss, J., Hogger, S., Oberhauser, C., & Tutz, G. (2011). Selection of ordinally scaled independent variables with applications to international classification of functioning core sets. Journal of the Royal Statistical Society: Series C (Applied Statistics), 60(3), 377–395.
- Gertheiss, J., & Tutz, G. (2009a). Penalized regression with ordinal predictors. International Statistical Review, 77(3), 345–365. Retrieved from http://epub.ub.uni-muenchen.de/2100/1/tr015.pdf.
- Gertheiss, J., & Tutz, G. (2009b). Supervised feature selection in mass spectrometry-based proteomic profiling by blockwise boosting. Bioinformatics, 25(8), 1076–1077.
- Gertheiss, J., & Tutz, G. (2009c). Variable scaling and nearest neighbor methods. Journal of Chemometrics, 23(3), 149–151. - Gertheiss, J. & Tutz, G. (2010). Sparse modeling of categorial explanatory variables. The Annals of Applied Statistics, 4, 2150–2180.
- Hofner, B., Hothorn, T., Kneib, T., & Schmid, M. (2011). A framework for unbiased model selection based on boosting. Journal of Computational and Graphical Statistics, 20(4), 956–971. Retrieved from http://epub.ub.uni-muenchen.de/11243/1/TR072.pdf.
- Oelker, M.-R., Gertheiss, J., & Tutz, G. (2012). Regularization and model selection with categorial predictors and effect modifiers in generalized linear models. Department of Statistics: Technical Reports, No. 122. Retrieved from http://epub.ub.uni-muenchen.de/13082/1/tr.gvcm.cat.pdf.
- Oelker, M.-R., & Tutz, G. (2013). A general family of penalties for combining differing types of penalties in generalized structured models. Department of Statistics: Technical Reports, No. 139. Retrieved from http://epub.ub.uni-muenchen.de/17664/1/tr.pirls.pdf.
- Petry, S., Flexeder, C., & Tutz, G. (2011). Pairwise fused lasso. Department of Statistics: Technical Reports, No. 102. Retrieved from http://epub.ub.uni-muenchen.de/12164/1/petry_etal_TR102_2011.pdf.
- Rufibach, K. (2010). An active set algorithm to estimate parameters in generalized linear models with ordered predictors. Computational Statistics & Data Analysis, 54(6), 1442–1456. Retrieved from http://arxiv.org/pdf/0902.0240.pdf?origin=publication_detail.
- Tutz, G. (2011, October). Regularization methods for categorical data. Munich: Ludwig-Maximilians-Universität. Retrieved from http://m.wu.ac.at/it/departments/statmath/resseminar/talktutz.pdf.
- Tutz, G., & Gertheiss, J. (2013). Rating scales as predictors—The old question of scale level and some answers. Psychometrika, 1-20.</sup>

Answer 2

When there are multiple predictors, and the predictor of interest is ordinal, it is often difficult to decide how to code the variable. Coding it as categorical loses the order information, while coding it as numerical imposes linearity on the effects of the ordered categories that may be far from their true effects. For the former, isotonic regression has been proposed as a way to address non-monotonicity, but it is a data-driven model selection procedure, which like many other data-driven procedures, requires a careful evaluation of the final fitted model and the significance of its parameters. For the latter, splines may partially mitigate the rigid linearity assumption, but numbers still must be assigned to ordered categories, and results are sensitive to these choices. In our paper (Li and Shepherd, 2010, Introduction, paragraphs 3-5), we gave a more detailed explanation of these issues, which are applicable to all regression models with an ordinal predictor of interest.

Let $Y$ be an outcome variable, $X$ be the ordinal predictor of interest, and $\bf Z$ be the other covariates. We have proposed to fit two regression models, one for $Y$ on $\bf Z$ and the other $X$ on $\bf Z$, calculate the residuals for the two models, and evaluate the correlation between the residuals. In Li and Shepherd (2010), we studied this approach when $Y$ is ordinal and showed that it can be a very good robust approach as long as the effect of the $X$ categories is monotonic. We are currently evaluating the performance of this approach on other outcome types.

This approach requires an appropriate residual for the regression of ordinal $X$ on $\bf Z$. We proposed a new residual for ordinal outcomes in Li and Shepherd (2010) and used it to construct a test statistic. We further studied the properties and other uses of this residual in a separate paper (Li and Shepherd, 2012).

We have developed an R package, PResiduals, which is available from CRAN. The package contains functions for performing our approach for linear and ordinal outcome types. We are working to add other outcome types (e.g., count) and features (e.g., allowing interactions). The package also contains functions for calculating our residual, which is a probability-scale residual, for various regression models.

References

Li, C. & Shepherd, B. E. (2010). Test of association between two ordinal variables while adjusting for covariates. JASA, 105, 612–620.

Li, C. & Shepherd, B. E. (2012). A new residual for ordinal outcomes. Biometrika 99, 473–480.

Answer 3

Generally there is lot of literature on ordinal variables as the dependent and little on using them as predictors. In statistical practice they are usually either assumed to be continous or categorical. You can check whether a linear model with the predictor as a continous variable looks like a good fit, by checking the residuals.

They are sometimes also coded cumulatively. An example would be for a ordinal variable x1 with the levels 1,2 and 3 to have a dummy binary variable d1 for x1>1 and a dummy binary variable d2 for x1>2. Then the coefficient for d1 is the effect you get when you increase your ordinal for 2 to 3 and the coefficient for d2 is the effect you get when you ordinal from 2 to 3.

This makes interpretation often more easily, but is equivalent to using it as a categorical variable for practical purposes.

Gelman even suggests that one might use the ordinal predictor both as a categorical factor (for the main effects) and as continous variable (for interactions) to increase the flexibility of the models.

My personal strategy is usually to look whether treating them as continous makes sense and results in a reasonable model and only use them as categorical if necessary.