Coding for Regression Analysis

Question

Q: What is the mathematical theory behind coding categorical variables for regression analysis?

The situation is this:

For numeric variables, we can code observations via the transformation

$\textrm{coded value} = \frac{\textrm{uncoded value} - \frac{(high+low)}{2}}{\frac{(high-low)}{2}}$.

For categorical variables, the above clearly will not work. I have been given the following codings without justification. For two variables, such as material type A and type B, we code them as

$ Type A = -1 \\ Type B = 1. $

If the there are three variables, we code them as

$ Type A = \{1,0\} \\ Type B = \{0,1\} \\ Type C = \{-1,-1\}. $

Once coded, we can then enter the data into SAS and run PROC REG to determine a regression model.

What is the mathematical basis for this coding? My textbook, Montgomery Design and Analysis of Experiments, 8th Edition, provides little more than a page on coding in general. My instructors cannot provide me with anything more than that it has something to do with orthogonality of vectors.

For the three variable case, we can see that any two vectors are linearly independent, but the three together are dependent. As such if we arrange them in a matrix, the matrix must then be singular.

Just as there is a justification for the numeric case, I want to understand the categorical. I'm not afraid of matrices or linear algebra, although it seems like there could be a simple geometric explanation. If it is too lengthy to explain, I would be happy with a textbook or online reference.

Answer 1

Consider your two types situation (say Gender to be concrete). Suppose, that you decide to code as follows:

If respondent is male: Set M = 1 or 0 otherwise

If respondent is female: Set F = 1 or 0 otherwise

Then the data set will look like so:

M F Y
1 0 y1
0 1 y2
1 0 y3
1 0 y4
0 1 y5
0 1 y6

Obviously, including both these variables as part of your $x$ matrix would lead to a singular design matrix which cannot be inverted. In other words, you cannot estimate the effect of both 'male' and 'female' respondents on your dependent variable independently of each other. Thus, you decide to drop one of these variables (say 'F'). Then assuming a model that looks like so:

$y=a + b M $

the OLS estimate of the intercept will give us effect of the average 'male' respondent on $y$ and $a+b$ the effect of the 'female' respondent. In other words, $b$ gives us the difference between female and male respondents on $y$.

What happens if we code M as $1, -1$?

Then the equation for male respondents and female respondents are:

$y_m = a + b$ and

$y_f = a - b$

Thus, $b$ is now $\frac{y_m-y_f}{2}$ which is bit complex to interpret. Thus, a simpler way to obtain interpretability is to choose a coding scheme such as: $0.5, -0.5$ instead of $1, -1$ which then results in:

$y_m = a + 0.5 b$ and

$y_f = a - 0.5 b$

Thus, $b$ is $y_m-y_f$ which is identical to the interpretation we obtained when we chose to code them as $1, 0$.

The above logic can be extended to the context of more than two categorical variables. If you want to know more you should probably look up 'Contrast Coding' as the different ways to code categorical variables are different ways to estimate contrasts between the categorical variables (e.g., do we want to estimate the impact of male - female in which case we pick the 'M' column or do we want to estimate the impact of female - male in which we case we pick the 'F' column).