Determining the Weight of Categorical Variable's Coefficient

Question

Lately, I have been studying about Logistic Regression, and I came across a question on how to handle categorical variable (as opposed to numerical ones).

Let's suppose I have a data table with two independent variables, and one dependent variable (the classification). Also, let's say one of the independent variables holds a numerical value, and the other one holds a categorical value. So the first one can be something like age, and the second one would be something like education level (high school, undergraduate, masters, phd).

To my knowledge, in order to perform logistic regression on this data set, I have to make the categorical variable into binary variable. So instead of 2 independent variables, I would have 5 independent variables (using the example I gave above, it would turn from [age, education] to [age, highschool, undergraduate, masters, phd] which the latter 4 would hold binary value.

Once logistic regression is done, it would give each variable a coefficient, a weight. Now in this case, I would have coefficient for each [highschool, undergraduate, masters, phd]. And here, I have a question:

In this case, would the 'weight' of the categorical variable (education) be the sum of 4 coefficients from [highschool, undergraduate, masters, phd] ? In other words, how would I measure the 'weight' (importance) of the categorical variable?

Answer 1

I would take a different approach than @IgorF. I would treat education as categorical.

In practice, in applications I am familiar with, your question is almost never answered. It is almost always an irrelevant question, and why waste time on something irrelevant? So my first suggestion to you is to explicitly formulate why you would want to know that, and why your audience might want to know that. Only once you have convinced yourself that it is worth your time and effort, start investing in it.

Once you have convinces yourself that this is a worthwhile exercise (remember that this is a very niche problem, so in all likelihood you will never reach this stage), then you could look at sheaf coefficients as a potential solution. See: (Heise 1972).

---

Some minor notes

Everything said here is not unique to logistic regression, it will apply equally to e.g. linear, poisson, probit, ordinal, multilevel regression.

You could argue education is ordinal in the US system. However, the way we treat ordinal variables as explanatory/independent/right-hand-side/x-variables is no different from categorical ones. So in practice it does not make a difference. Moreover, in more tracked systems the ordinal interpretation of education gets messy pretty quickly: how do you compare a complete lower track with an incomplete higher track?

If you have a four category explanatory variable, then you create three indicator (dummy) variables. One of the categories is treated as the reference category.

As description of the American education system you are missing a hugely important category: less than highschool, and the PhD is usually too small a group to be really worthwhile measuring.

Heise, David R. (1972). Employing nominal variables, induced variables, and block variables in path analysis. Sociological Methods & Research, 1(2): 147--173. https://doi.org/10.1177%2F004912417200100201

---

Response to comment

There was some interest in my field in the comparison of weights in the 1960's and 1970's, but it is now largely considered (very) outdated. The reason becomes clear when you look more closely what the weights actually mean. In case of logistic regression: if you get a unit more of x then the odds of the outcome increases by a factor $\exp(\beta)$. So taking your example from your comment: if you get a year older then the odds of the outcome changes by a factor $\exp(\beta_1)$, and lets say your nationalities are Belgium, the Netherlands, and Luxembourg, and the Netherlands is your reference category. In that case you would interpret $\exp(\beta_2)$ as the odds of the outcome is $\exp(\beta_2)$ times bigger/smaller in Belgium than in the Netherlands. The reason it is hard to compare these weights is that the units are completely different: how would you compare the difference between a year and Belgium versus the Netherlands? Attempts have been made to find a common metric, like the sheafcoefficient, but you can imagine that the result is highly abstract. As a consequence you typically loose much more than you gain by these methods: we know what getting a year older means, but what an unitless number might mean is much harder (plus all the assumptions you need to believe in order to justify that these numbers are really comparable). So nowadays we just focus on the actual meaning of the weights and the story we can tell with them, rather than artificially ranking predictors from most to least important.

Answer 2

Your "education" variable is not simply categorical, it is ordinal. High school is more than no school, undergraduate school is more than nothing or high school etc. Ordinal variables can be thought of as discretised, observable effects of some underlying latent variable.

In many cases, you can treat ordinal variable as numeric. E.g. no school = 0, high school = 1, ..., PhD = 4. You can argue that this is not entirely correct, as the "skill difference" between PhD and masters is not as large as between high school and no school. That's true, but, you wouldn't be better off if you had the access to the latent variabel, but the dependency between it and the outcome having some unknown non-linearity. In other words, the error you make by treating an ordinal variable as numeric is not different from assuming linearity when the dependency is non-linear.

In your case, you can even do better. The education level can easily be translated into years of education, which is numeric: High school = 12, undergraduate = 15-16, etc. That simplifies your problem to two numeric predictors.