1
$\begingroup$

Let us consider first case. I have logistic regression with one categorical variables with 3 distinct values. I can code as

Value A -> (0,0)
Value B -> (1,0)
Value C -> (0,1)

And with intercept my variables would look like

Value A -> (1,0,0)
Value B -> (1,1,0)
Value C -> (1,0,1)

Now once I have regression done, it is easy to interpret coefficients of regression. Value A will serve as base case. Let $x_B$ denote vector of value where B outcome of our categorical variable is present (and so on for other outcomes) and $\beta_B$ is the coefficient of B value

$odds(x):=p(x)/(1-p(x))$

$\frac{e^{x_B}}{e^{x_A}} = e^{\beta_B} = \frac{odds(x_B)}{odds(x_A)}$

So interpretation of $\beta_B$ is log of ratio of odds of B to A.

Now let us consider another case. I have two categorical variables. First variable is $Z$ with values $A,B,C$ and another categorical variable is $W$ with values $E,F,G$.

Now let us try to code it similar to the first case (with intercept)

value A -> 1,0,0,0,0,0
value B -> 1,1,0,0,0,0
value C -> 1,0,1,0,0,0
value E -> 1,0,0,1,0,0
value F -> 1,0,0,0,1,0
value G -> 1,0,0,0,0,1

With this coding I no longer get the same simplification as in the first case and interpret $\beta_E$ as before. Simplification works, but I get to compare value A and value E from different variables, while I rather compare outcomes within one variable.

Is there workaround, am I missing something?

UPDATE

Here is example using R to clarify my question

x <- sample(c('green', 'blue', 'red'), size=20, replace= TRUE)
y <- sample(c('east', 'west', 'north'), size=20, replace= TRUE)
outcome <- sample(c(1,0), size=20, replace= TRUE)
my_df <- data.frame(x, y, outcome)

model <- glm(outcome~., family=binomial(link = 'logit'), data=my_df)
summary(model)

Print out of results

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)  -0.3449     1.2934  -0.267   0.7897  
xgreen       -2.9008     1.6541  -1.754   0.0795 .
xred         -1.5273     1.4097  -1.083   0.2786  
ynorth        1.7580     1.5611   1.126   0.2601  
ywest         3.3712     1.8804   1.793   0.0730 .

What are interpretations of coefficienst and how interpretations are derived?

$\endgroup$
  • 1
    $\begingroup$ Are you using any program to perform your analyses? Most of the modern statistical programs take care of all of this coding for your automatically. Also, it's not 100% clear to me what you are asking. Do you want to calculate the odds ratio of B to A with this new, expanded model, but you are confused as to how to do so because of the added complexity of having to average over other independent variables? $\endgroup$ – StatsStudent May 30 '17 at 19:55
  • $\begingroup$ @StatsStudent: Let me add example using R to clarify my question. $\endgroup$ – user1700890 May 30 '17 at 20:02
  • 1
    $\begingroup$ There is an important and illustrative mismatch between your presentation of the model matrix construction (following the text "Now let us try to code it similar to the first case (with intercept)"), in which your matrix has six columns, and the R output, which gives only five parameter estimates. Your coding actually corresponds to a single categorical variable with possible values A,B,C,E,F,G, rather than two categorical variables. If the distinction is still unclear, please execute model.matrix(outcome~., my_df) and inspect the output closely. $\endgroup$ – whuber May 30 '17 at 20:21
  • 1
    $\begingroup$ @whuber Thank you, what a great command model.matrix(outcome~., my_df)! I wish I knew it before I asked the question. $\endgroup$ – user1700890 May 30 '17 at 21:49
4
$\begingroup$

I see a few conceptual mistakes here.

First of all, if $Z$ and $W$ are two variables in logistic regression, there should be no constraint on their input value. Logistic regression is a discriminative model, so no assumption is made about the distribution of the input variables. Therefore the values of $Z$ and $W$ should not be mutually exclusive. You are allowed to have an entry for both, which you neglected to allow when listing your values. Thus you have nine possible entries, not six.

Secondly, as you noted with one variable, each categorical variable need only one less binary variable than the number of classes. So with two categorical variables, each with three classes, you have 5 inputs. One bias (which is always one), and four binary variables representing the two 3-class categorical variables.

We can look at the model as having five parameters. $\beta_B,$ $\beta_C,$ $\beta_F,$ $\beta_G,$ and $\beta_0,$ where each subscript represents the categorical value it represents, and $\beta_0$ is the bias term which represents the "reference" values $Z=A$ and $W=E.$ (This is an arbitrary choice.) In general, the bias term absorbs the reference value of every categorical variable in the model.

Now, let's denote $l = \beta' X$ as the log-odds of an observation given $X$ (where $X$ is a five-vector of binary variables representing an observation and $\beta$ is the vector of five parameters). (Note that by "odds of an observation", what I mean is that it's the odds ratio of a "one" observation of your target variable given $X$ with respect to a "zero" observation given $X$.) Suppose we have $X_1 = (1,0,0,0,0)'$ and $X_2 = (1,0,0,1,0)'.$ This corresponds to $(Z,W) = (A,E)$ and $(Z,W) = (A,F)$ respectively. As well, $l_1 = \beta' X_1$ and $l_2 = \beta' X_2.$ Then, $$ \frac{\text{odds}(X_2)}{\text{odds}(X_1)} = e^{l_2 - l_1} = e^{\beta_F}. $$

So $\beta_F$ is the log of the odds ratios between observations $(Z,W) = (A,F)$ and $(Z,W) = (A,E).$ However, it's easy to see that this relationship would hold independent of what the value of $Z$ is. That is, if $Z=Z_0,$ then using the same analysis, we would find that $\beta_F$ is the log of the odds ration between observations $(Z_0,F)$ and $(Z_0,E),$ no matter what the value of $Z_0.$ Thus, $\beta_F$ is the log of the odds ratio between $W=F$ and $W=E.$ This is simply because of the choice of $E$ as the base line value.

You can easily extend this argument to any value of a categorical variable, with respect to the choice of reference value. In general, if $Z$ is a categorical input variable in logistic regression with $k$ levels, then it contains $k-1$ parameters, each of which represents the log-odds of an observation given a particular value of $Z$ with respect to an observation given the reference value of $Z.$ This is true regardless how however many other categorical variables are in the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.