# Multivariate Regression on Multicollinear Categorical Variables

I am working on a dataset with a continuous response (which could be dichotomized), one continuous covariate, and multiple categorical variables. The continuous covariate (weight) is directly correlated to the response, and must be accounted for so that we can determine which of the categorical variables are most influential to the response. Here is example data.

Each row is an individual subject, with the continuous response, the covariate of underlying primary importance (weight), then 10 categorical variables that are to be tested (individuals can score yes = 1 to multiple categories).

My first thought in working with this data was a linear model, with stepwise elimination of categorical variables.

 lm(Response~Weight+var1+var2...+var11)


However, I believe there is extensive collinearity, since some variables may be eliminated early, but then are significant if you add them back into the model at the end. I'm curious if there is a better way to approach this data in R, that may help sort through which of the variables are of most importance to influencing the response. My two thoughts are

1) Building a single model with the continuous covariate and 5 categorical variables that were selected to be of most interest before the study, and refrain from any stepwise reduction of this model

2) Some sort of princicpal component regression, which I know little about at this point and thus wanted to ask advice before proceeding down that path

To help visualize the data, and the effect of Weight on the Response, I've constructed the follow plots. In the second plot, I attempt to control for the natural Response~Weight relationship.

 #GRAPH
library(ggplot2)
library(reshape2)

#Creating long format for ggplot2
Data2<-melt(Data, id.vars = c("Subject","Response","Weight"), measure.vars = c("var1","var2","var3","var4","var5","var6","var7","var8","var10","var11"))

#Adding in weight to the varibles to be plotted
Data2<-rbind(Data2,Data2[1:31,])
levels(Data2$variable)<-c(levels(Data2$variable),"Weight")
Data2[311:341,4]<-"Weight"
Data2[311:341,5]<-1

#Removing rows where the categorical variable is 0=No
for(i in 1:length(Data2[,1])){
if(Data2[i,5]==0)Data2[i,]<-NA
}
Data3<-na.omit(Data2)

#Plotting Response vs Weight for each 'Yes' group for the categorical variables
scatter <- ggplot(Data3, aes(Weight, Response, colour = variable))
scatter + geom_point(aes(color = variable), size = 3) + geom_smooth(method = "lm",aes(fill = variable), alpha = 0.1) + facet_wrap(~variable)+ guides(fill=FALSE,color=FALSE)


 #Zeroing the Response~Weight relationship to remove its influence. Correction coefficients from linear model fit to Response~Weight
Data4<-Data3
Data4$Response<-Data4$Response-(0.01494*(Data4\$Weight)+ 84.67715)

#Plotting Response vs Weight for each 'Yes' group for the categorical variables for zeroed Response~Weight relationship (as seen in bottom right facet)
scatter2 <- ggplot(Data4, aes(Weight, Response, colour = variable))
scatter2 + geom_point(aes(color = variable), size = 3) + geom_smooth(method = "lm",aes(fill = variable), alpha = 0.1) + facet_wrap(~variable)+ guides(fill=FALSE,color=FALSE)


This second plot helps to show how, when the Response~Weight relationship is controlled for, variables like 'var10' have no influence on the response, while variables like 'var11' have all individuals below that zero-centered mean. Thus, from a visual test, I could identify var11 as a categorical variable of interest that negatively influences our response.

Additionally, this plot shows some of the confounding in this dataset, as you can see certain categorical variables 'clump'/are only documented in certain weight ranges. This is due to the underlying biology.

As a final note, I wonder if it is appropriate to use the corrected response in the second plot as the 'Response' for a linear model, thus eliminating the need for a 'Weight' covariate, or if it is incorrect to use such a transformation

Any thoughts are much appreciated

• voting to close/send to CrossValidated ... – Ben Bolker Oct 23 '14 at 14:17
• You have accepted an answer that didn't properly point out the errors in this approach. MC is not a problem with relationships of predictors to response. – DWin Oct 23 '14 at 17:50
• I agree that Option 1 in the below answer is not appropriate. However, would you agree that running vif(lm(Response~Weight+var1+var2...+var11)), and then removing 2 of the 3 MC variables to create a new lm() is appropriate? – user3708129 Oct 23 '14 at 18:11

This question is for CrossValidated but here are a few suggestions:

1) Check multi-collinearity using VIF in the cars package between variables. VIF > 10 is not desirable.

If Multi Collinearity exists there are three options:

(1) Check the relationship of the variables with the response to understand what kind of transformation can be applied e.g. quadratic, polynomial, logarithmic etc.

(2) Use PCA to combine some of the variables. Here is how you can reproduce that in R : https://stackoverflow.com/questions/18139292/reproducing-spss-factor-analysis-with-r/25070213#25070213

(3) Remove the multi-collinear variables. Essentially the probability of a multi collinear impacting the outcome is low. you can observe this in the t-value of the variable. There are "*" symbols in the Pr(>|t|) field of the model output. It denotes the significance or the likelihood of the variable to not be influencing the response.

• Strategy 1 is not going to be helpful assessing or managing multicollinearity, although it is a step one should always perform and plotting methods are often very informative. The MC problem is in the joint relationship among the predictors, not with their relationship to the response. – DWin Oct 23 '14 at 17:47
• Indeed you are correct, I should have posted to CrossValidated for this issue. However, the vif() function has been very helpful at identifying 3 potential collinear variables. I have approached this with Option 3, and have removed the two collinear variables which were of least importance – user3708129 Oct 23 '14 at 18:09