1
$\begingroup$

Consider the following MWE example. A dataset with only one feature (categorical feature with 4 different categories ['cat', 'dog', 'hamster', 'frog']) + target (overall 10% positive class). After ohe and dropping one of the 4 resulting columns, e.g.,

is_cat    is_dog    is_hamster   Target
  1          0          0           1
  0          1          0           0
  0          0          1           1
  0          1          0           0
  1          0          0           0
.
.
.
 1           0          0           0

Let's suppose also that the average target for each category is 'cat':15%, 'dog':10%, 'hamster':8%. Now, consider a Logistic Regression $log(\frac{p}{1-p}) = \beta_0 + \sum_i \beta_i x_i$ fitted with the data above. The obtained estimates are:

$\beta_0=-1.53$, $\beta_{dog}=-0.65$, $\beta_{cat}=-0.2$, $\beta_{hamster}=-0.9$

However, I would expect the estimates on the log scale to be

negative for hamster (decrease the odds)
zero for dogs (no difference with the baseline)
positive for cats (increase the odds) 

However, the 3 coefficients for the fitted curve show a negative sign, so either the approach or the interpretation is not correct. Is that the right way to interpret the coefficients?

$\endgroup$
5
  • $\begingroup$ I assume you are talking about negative estimates on the log-odds scale? If you exponentiate them, you will get odds ratios, which should be easier to interpret. $\endgroup$ Commented Jun 1, 2021 at 18:25
  • $\begingroup$ Exactly. Edited the question for clarity. I think the intercept term is the key here, since negative estimates on the log-odds scale will decrease the final predicted probability, but I noticed the sigmoid(intercept) does not correspond with the baseline. Is there a way of interpreting the sign of the estimates then? $\endgroup$
    – simon
    Commented Jun 1, 2021 at 18:29
  • 1
    $\begingroup$ Perhaps if you could edit the question again and add the output from the model, it will be easier to help. Otherwise you try a search of this site for questions about interpreting logistic regression. $\endgroup$ Commented Jun 1, 2021 at 18:38
  • $\begingroup$ Are there any cases that aren't dog, cat or hamster? You may be running into a problem otherwise. If all animals are cats, dogs or hamster, you only need two variables. What do you get if you remove is_dogfrom the model? $\endgroup$
    – David
    Commented Jun 1, 2021 at 19:06
  • $\begingroup$ Yes, there was a 4th category, which is not included here. Amended the question. $\endgroup$
    – simon
    Commented Jun 1, 2021 at 19:11

1 Answer 1

2
$\begingroup$

The beta for (e.g.) hamster isn't the difference from the log odds of the base rate, it's the difference from the log odds for the reference level (i.e., frog). Without knowing what proportion of the critters are each level, we can't compute what they should be, but I see no reason to suspect the outputted values are wrong.

$\endgroup$
6
  • $\begingroup$ I get it. The reference level is the positive class rate for the unseen category (i.e., a column with all 0es), not the overall positive class rate in the data. How would it help to know the proportion of the critters? (say, e.g. 1/4 of each). $\endgroup$
    – simon
    Commented Jun 1, 2021 at 19:34
  • $\begingroup$ If I knew the proportions, we could calculate the differences independently of the model fit. (Ie, as a basic gut check.) $\endgroup$ Commented Jun 1, 2021 at 19:35
  • $\begingroup$ I see. So in a general, in a more complex setting with tens of features where the reference level will not be explicit, is it possible to make any further statement based only on the sign of the betas? $\endgroup$
    – simon
    Commented Jun 1, 2021 at 20:10
  • $\begingroup$ The coefficients for the levels of a categorical variable are essentially a set of intercepts. If you have no interactions, they are equal vertical shifts everywhere, as the lines will be parallel (& in a GLiM, straight on the scale of the linear predictor). In general, the coefficient for a specified level is the difference (here, in log odds) between the indicated level and the reference level. That's true whether or not there are continuous covariates, & no matter how many categorical variables you have or how many levels they have. $\endgroup$ Commented Jun 1, 2021 at 20:18
  • $\begingroup$ How does that connect with the odds ratio interpretation of the coefficient? In general, a negative beta for a specific level means the odds decrease with the level, which Im struggling to understand in the case of e.g. the 'cat' level in the question $\endgroup$
    – simon
    Commented Jun 1, 2021 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.