Qualitative variable coding in regression leads to "singularities" - Cross Validated most recent 30 from stats.stackexchange.com 2019-11-22T16:02:17Z https://stats.stackexchange.com/feeds/question/70699 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/70699 17 Qualitative variable coding in regression leads to "singularities" varin sacha https://stats.stackexchange.com/users/25335 2013-09-22T15:07:00Z 2017-07-12T10:37:59Z <p>I have an independent variable called "quality"; this variable has 3 modalities of response (bad quality; medium quality; high quality). I want to introduce this independent variable into my multiple linear regression. When I have a binary independent variable (dummy variable, I can code <code>0</code> / <code>1</code>) it is easy to introduce it into a multiple linear regression model.</p> <p>But with 3 modalities of response, I have tried to code this variable like this :</p> <pre><code>Bad quality Medium quality High quality 0 1 0 1 0 0 0 0 1 0 1 0 </code></pre> <p>But there is a problem when I try to do my multiple linear regression: the modality <code>Medium quality</code> gives me <code>NA</code>: </p> <pre><code>Coefficients: (1 not defined because of singularities) </code></pre> <p>How can I code this variable "quality" with 3 modalities? Do I have to create a variable as a factor (<code>factor</code> in <code>R</code>) but then can I introduce this factor in a multiple linear regression?</p> https://stats.stackexchange.com/questions/70699/-/70700#70700 23 Answer by gung - Reinstate Monica for Qualitative variable coding in regression leads to "singularities" gung - Reinstate Monica https://stats.stackexchange.com/users/7290 2013-09-22T15:33:53Z 2013-09-22T16:00:58Z <p>The problem you are having (i.e., "singularities") can be thought of as an instance of <a href="http://en.wikipedia.org/wiki/Multicollinearity" rel="noreferrer">multicollinearity</a>. Multicollinearity is often defined as: </p> <blockquote> <p>One or more predictor variables are a linear combination of other predictor variables. </p> </blockquote> <p>This is, in fact, a rather strict definition; it is <em>perfect</em> multicollinearity, and you can easily have a problem with multicollinearity without any of your variables being <em>perfect</em> linear combinations of others. Moreover, perfect multicollinearity rarely occurs. However, you have stumbled across an case where it can occur. Let us see how we can <em>perfectly</em> predict <code>medium quality</code> from our knowledge of the other two categories (we'll do this with a regression model where <code>medium quality</code> is $Y$, and <code>bad quality</code> &amp; <code>high quality</code> are $X_1$ &amp; $X_2$, respectively):<br> $$Y = \beta_0 + \beta_1X_1 + \beta_2X_2$$ Note that there is no error term, $\varepsilon$, specified, because we can predict this perfectly. To do so, we set $\beta_0 = 1$, $\beta_1 = -1$, and $\beta_2 = -1$. Now, when you have <code>bad quality</code>, then $X_1=1$, which cancels out $\beta_0$ ($1\; + \;-1\!\times\! 1$), and $X_2=0$ so that term is canceled out as well ($-1\times 0$). Thus, we are left with a predicted value of $0$ for $Y$ (<code>medium quality</code>), which is exactly correct. I will leave it to you to work out the other possibilities (it always works, in your case). </p> <p>So what then should you do? When representing a categorical variable, we typically use <em>reference cell coding</em> (often called 'dummy coding'). To do this, we pick one level of our categorical variable as the reference level; that level does not get its own dummy code, but is simply indicated by having all $0$'s in the dummy codes for all other levels. The other levels of your categorical variable are represented by dummy codes just as you have already done. (For some more information on this, you can see my answer here: <a href="https://stats.stackexchange.com/questions/21282//21292#21292">Regression based for example on days of week</a>.) If you are using <code>R</code>, you can use a <code>factor</code> and <code>R</code> will do this all for you--it will be done correctly, and it's much more convenient--nonetheless, it's worth understanding that this is what is happening 'behind the scenes'. </p> https://stats.stackexchange.com/questions/70699/-/70708#70708 10 Answer by dardisco for Qualitative variable coding in regression leads to "singularities" dardisco https://stats.stackexchange.com/users/11756 2013-09-22T17:38:46Z 2013-09-22T17:38:46Z <p>@gung has explained the theory clearly. Here's a practical example to illustrate:</p> <pre><code>set.seed(1) pred1 &lt;- factor(c("bad", "med", "high"), levels=c("bad", "med", "high")) df1 &lt;- data.frame(y=20*abs(runif(6)), x=rnorm(6), q=sample(pred1, 6, replace=TRUE) ) l1 &lt;- lm(y ~ x, data=df1) ### add variable q l2 &lt;- lm(y ~ x + q, data=df1) ### look at dummy variables generated in creating model model.matrix(l2) </code></pre> <p>This shows us that the reference level (all $0$s) is <code>bad</code> as seen here in row 4:</p> <pre><code> (Intercept) x qmed qhigh 1 1 1.5952808 1 0 2 1 0.3295078 0 1 3 1 -0.8204684 0 1 4 1 0.4874291 0 0 5 1 0.7383247 1 0 6 1 0.5757814 0 0 </code></pre> <p>Now if we code the dummy variables ourselves and try to fit a model using all of them:</p> <pre><code>df1 &lt;- within(df1, { qbad &lt;- ifelse(q=="bad", 1, 0) qmed &lt;- ifelse(q=="med", 1, 0) qhigh &lt;- ifelse(q=="high", 1, 0) }) lm(y ~ x + qbad + qmed + qhigh, data=df1, singular.ok=FALSE) </code></pre> <p>We get the expected error: <code>singular fit encountered</code></p>