3
$\begingroup$

Lately, I have been studying about Logistic Regression, and I came across a question on how to handle categorical variable (as opposed to numerical ones).

Let's suppose I have a data table with two independent variables, and one dependent variable (the classification). Also, let's say one of the independent variables holds a numerical value, and the other one holds a categorical value. So the first one can be something like age, and the second one would be something like education level (high school, undergraduate, masters, phd).

To my knowledge, in order to perform logistic regression on this data set, I have to make the categorical variable into binary variable. So instead of 2 independent variables, I would have 5 independent variables (using the example I gave above, it would turn from [age, education] to [age, highschool, undergraduate, masters, phd] which the latter 4 would hold binary value.

Once logistic regression is done, it would give each variable a coefficient, a weight. Now in this case, I would have coefficient for each [highschool, undergraduate, masters, phd]. And here, I have a question:

In this case, would the 'weight' of the categorical variable (education) be the sum of 4 coefficients from [highschool, undergraduate, masters, phd] ? In other words, how would I measure the 'weight' (importance) of the categorical variable?

$\endgroup$

2 Answers 2

1
$\begingroup$

I would take a different approach than @IgorF. I would treat education as categorical.

In practice, in applications I am familiar with, your question is almost never answered. It is almost always an irrelevant question, and why waste time on something irrelevant? So my first suggestion to you is to explicitly formulate why you would want to know that, and why your audience might want to know that. Only once you have convinced yourself that it is worth your time and effort, start investing in it.

Once you have convinces yourself that this is a worthwhile exercise (remember that this is a very niche problem, so in all likelihood you will never reach this stage), then you could look at sheaf coefficients as a potential solution. See: (Heise 1972).


Some minor notes

Everything said here is not unique to logistic regression, it will apply equally to e.g. linear, poisson, probit, ordinal, multilevel regression.

You could argue education is ordinal in the US system. However, the way we treat ordinal variables as explanatory/independent/right-hand-side/x-variables is no different from categorical ones. So in practice it does not make a difference. Moreover, in more tracked systems the ordinal interpretation of education gets messy pretty quickly: how do you compare a complete lower track with an incomplete higher track?

If you have a four category explanatory variable, then you create three indicator (dummy) variables. One of the categories is treated as the reference category.

As description of the American education system you are missing a hugely important category: less than highschool, and the PhD is usually too small a group to be really worthwhile measuring.

Heise, David R. (1972). Employing nominal variables, induced variables, and block variables in path analysis. Sociological Methods & Research, 1(2): 147--173. https://doi.org/10.1177%2F004912417200100201


Response to comment

There was some interest in my field in the comparison of weights in the 1960's and 1970's, but it is now largely considered (very) outdated. The reason becomes clear when you look more closely what the weights actually mean. In case of logistic regression: if you get a unit more of x then the odds of the outcome increases by a factor $\exp(\beta)$. So taking your example from your comment: if you get a year older then the odds of the outcome changes by a factor $\exp(\beta_1)$, and lets say your nationalities are Belgium, the Netherlands, and Luxembourg, and the Netherlands is your reference category. In that case you would interpret $\exp(\beta_2)$ as the odds of the outcome is $\exp(\beta_2)$ times bigger/smaller in Belgium than in the Netherlands. The reason it is hard to compare these weights is that the units are completely different: how would you compare the difference between a year and Belgium versus the Netherlands? Attempts have been made to find a common metric, like the sheafcoefficient, but you can imagine that the result is highly abstract. As a consequence you typically loose much more than you gain by these methods: we know what getting a year older means, but what an unitless number might mean is much harder (plus all the assumptions you need to believe in order to justify that these numbers are really comparable). So nowadays we just focus on the actual meaning of the weights and the story we can tell with them, rather than artificially ranking predictors from most to least important.

$\endgroup$
3
  • $\begingroup$ Thanks for the answer. If I consider another case, in which variable is a categorical variable like nationality, I may be wondering if a person's age has more weight on the outcome of classification or his/her nationality has more weight. In this case I may wonder if how much the factor of nationality plays into classification. $\endgroup$
    – Joon
    Mar 11, 2021 at 4:17
  • $\begingroup$ (Assuming that this is one of the cases where it is relevant) $\endgroup$
    – Joon
    Mar 11, 2021 at 4:18
  • $\begingroup$ @Joon I have edited my answer to clarify that to answer your question you need to find a way to compare apples and oranges. This is possible (they are both fruit), but in doing so you necessarily also loose something. My rule of thumb would be to assume that this question is not relevant, and require some really really really strong arguments to move away from that assumption. If you have that strong argument you can look at the sheafcoeficient, as I mentioned in the answer. $\endgroup$ Mar 11, 2021 at 8:53
0
$\begingroup$

Your "education" variable is not simply categorical, it is ordinal. High school is more than no school, undergraduate school is more than nothing or high school etc. Ordinal variables can be thought of as discretised, observable effects of some underlying latent variable.

In many cases, you can treat ordinal variable as numeric. E.g. no school = 0, high school = 1, ..., PhD = 4. You can argue that this is not entirely correct, as the "skill difference" between PhD and masters is not as large as between high school and no school. That's true, but, you wouldn't be better off if you had the access to the latent variabel, but the dependency between it and the outcome having some unknown non-linearity. In other words, the error you make by treating an ordinal variable as numeric is not different from assuming linearity when the dependency is non-linear.

In your case, you can even do better. The education level can easily be translated into years of education, which is numeric: High school = 12, undergraduate = 15-16, etc. That simplifies your problem to two numeric predictors.

$\endgroup$
2
  • 1
    $\begingroup$ That is a very American way of interpreting the value of education. That is fine, as the categories specified by the OP suggest that her/his data refer to American data. Just remember that this does not work for other educational systems. Especially in systems with a strong occupational education track, the number of years in school has little or no relationship with the "level" or "value" of a school level. $\endgroup$ Mar 10, 2021 at 13:28
  • $\begingroup$ I can see how for the case I gave above, the education level could be translated into numerical values. But if I have a variable such as "Region" or "Nationality", this can't be translated into numerical values. I was wondering how I would handle cases like this. $\endgroup$
    – Joon
    Mar 11, 2021 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.