Dummy Coding for Regression?

Question

I'm confused about which predictor should have the "0" and which should have the "1" when using dummy codes for regression. For example:

Y: time spent at current job
X: type of assessment --> interview (I), competency exam (C), written test (W)

Question: a researcher is interested in comparing differences in time spent at a current job (Y) between those given an interview (X1) and those given competency exams (X2).

I designed my table like this:

1. New variable 1: I --> +1, C --> 0, W --> 0 (Because I am comparing competency individually to interview)
2. New variable 2: I --> 0, C --> 0, W --> +1 (Because I am comparing competency individually to written)

However, these new variables are not orthogonal. Is that OK? (I'm not sure I understand what the point of variables being orthogonal is. I'm so lost. Also, when do I use dummy coding instead of contrast coding? Is there a hard and fast rule for when to default to either?

Answer 1

Your variables are correct, but your reasoning is strange. You need one less binary than you have categories. Each binary represents the presence of one category vs the rest of possible categories. If all but one categories are absent, it must be the last one. The last category can be seen as the baseline. This is the intuitive way of thinking about this. You can in principle also use +1 -1 binaries for regression but it's less intuitive.

Many software packages automate this process as soon as you tell them a regressor is categorical.

Answer 2

I don't understand your table design at all, that's just me.

Now in regards to figuring out the way to model indicator variables: Rather than hand-code the design matrix you can try to use software like R and supply the model there.

You can then extract the design from R and have a look at how R models dummy variables.

You'll notice that if you included an intercept in the model the first column of the design is a column of all 1's. Then in case where you have two indicators you will see that column two in the design matrix is 0s for those measurements that correspond to the most part and 1's for the rest.

Try something like (not copy-pastable):

linear1 = lm(Y~worktype, data=Yourdata)
model.matrix(linear1)

and R will plot the design matrix where you can inspect it. I personally found it instructive to plot it, and then try to fiddle around with different encoding values to figure out why one indicator has to be modeled as 0s for this particular encoding scheme.

Note that there are several different encoding schemes (I guess you knew that already) and you can find some here: http://www.ats.ucla.edu/stat/sas/webbooks/reg/chapter5/sasreg5.htm. It seems to me, you've focused on one that is rather more difficult to work with, given the simplicity of the data you're looking at.

Orthogonal Variables Variables need to be orthogonal because linear regression projects data points into a space spanned by your variables. A projection matrix is among other things, identified by being idempotent and by having eigenvalues equal to 0 or 1. One can ponder why that must be the case for a projection. That should help develop geometric intuition as well as linear regression intuition. I think, personally, that one must realize that linear regression is a projection from an $n$ dimensional space into a $p$ dimensional space spanned by your variables. General linear regression projects orthogonally into your variable space.

@Whuber has on several occasions described what this looks and feels like all the way from intuition to mathematics that made my head spin... I heartily recommend searching his posts on linear regression to see if something instructive doesn't pop up. Hint: Something instructive pops up.

I'm harping on a bit about this because the geometry of general linear regression should be something where one could be woken up at 3 AM, be pop-quizzed on, and not fail :) It also gives a sense of how to interpret 'significant' variables and the issues that arise in data sets that have very many variables.

Now, back to orthogonality: If you cannot uniquely determine where a data point is projected you have more than one way to describe the projection and thus more than one way to establish what $\hat{\beta}$ could be when calculating $Y = X\hat{\beta} + \epsilon$. If that is the case, you need rules on how to find the 'best' parameter estimates. The science of finding and applying these rules is called regularization.

In simpler words: the values that go into the beta vector describe the effect the given task has on working time, but if the categorical values are not modeled as orthogonal, you cannot deduce what the effect of the task on working time really is, because you in effect have several answers that (without expert knowledge on the subject) could potentially all be correct.

Edit: What does orthogonal mean in the context of statistics? A good place to start reading. A user actually addresses orthogonality of factors in one of the answers.