3
$\begingroup$

Here is an example model using diamonds dataset:

library(tidyverse)
library(caret)
library(broom)

# data
my_diamonds <- diamonds
my_diamonds <- my_diamonds %>% 
  mutate(target = ifelse(cut == "Fair" | cut == "Good", 0, 1))


# statistical modeling with logistic regression via caret
## tuning & parameters
set.seed(123)
train_control <- trainControl(
  method = "cv",
  number = 5,
  savePredictions = TRUE,
  verboseIter = TRUE,
  classProbs = TRUE,
  summaryFunction = prSummary
)

# fit model
linear_model = train(
  x = select(my_diamonds, c(price, x, y, z)), 
  y = my_diamonds$target %>% make.names() %>% factor(levels = c("X1", "X0")),
  trControl = train_control,
  method = "glm", # logistic regression
  family = "binomial",
  metric = "AUC" # actually prAUC
)

# get coefficients and various ratios
model_df <- broom::tidy(linear_model$finalModel) %>% 
  select(term, estimate) %>% 
  mutate(odds_ratio = exp(estimate))

Here is the output of model_df: model_df

# A tibble: 5 x 3
  term         estimate odds_ratio
  <chr>           <dbl>      <dbl>
1 (Intercept) -5.81        0.00299
2 price       -0.000201    1.000  
3 x           -1.72        0.179  
4 y           -2.45        0.0867 
5 z            7.99     2959.     

Within the context of inference and understanding, can I infer the following:

  1. For inference, the estimate is not so interesting, it's the logit which is scaled log odds between 0 and 1. Typically, of more interest is the log odds... True or False?

  2. The log odds, calculated above with exp(estimate) are the change in odds relative to the intercept of each coefficient. Example, in this model price has a log odds ratio of exactly 1, so the change in odds per one unit change in price is unchanged, since it changes by a factor of 1 only. i.e. if price is 10 then the odds of target being 1 are still just 0.00299 all else being equal. If the log odds for price were 2 and not 1, then the odds of target being 1 when price is 10 would be (0.00299*2)^10. Is this correct?

  3. Each time input variable x increases by one, the log odds of target being 1 increases by 0.179 * -1.72 = −0.30788.

  4. For understanding actual probabilities, I was initially tempted to do this:

# don't do this because the probability ratio is not linear, must be applied to a given observation not the coefficients model_df_dont_do_this <- model_df %>% mutate(probs_ratio = exp(estimate)/(1 + exp(estimate)))

Looks like this:

model_df_dont_do_this
# A tibble: 5 x 4
  term         estimate odds_ratio probs_ratio
  <chr>           <dbl>      <dbl>       <dbl>
1 (Intercept) -5.81        0.00299     0.00298
2 price       -0.000201    1.000       0.500  
3 x           -1.72        0.179       0.152  
4 y           -2.45        0.0867      0.0798 
5 z            7.99     2959.          1.000  

However, from researching on this site and elsewhere, this is nonsensical since the probability ratio is not linear and can only be applied to a given observation, not to the coefficients. Is that correct?

I'm seeking validation on my understanding. Have I understood logistic regression correctly? If not, where have I misunderstood?

$\endgroup$
1
$\begingroup$

You are correct in your statement "this is nonsensical since the probability ratio is not linear and can only be applied to a given observation, not to the coefficients". Even what you call "odds ratio" are not quite coefficients, though they are useful

I cannot actually get your linear_model to work, though

fit1 <- glm(target ~ price + x + y + z, family=binomial, data=my_diamonds)
fit1$coefficients
exp(fit1$coefficients)
exp(fit1$coefficients) / (1 + exp(fit1$coefficients))

produces the same coefficients as you have up to rounding, though with the "log-odds coefficients" signs reversed, the "odds coefficients" as the reciprocals of yours, and the "probability coefficients" $1$ minus your values (probably something to do with the translation to factors and their use, so your log-odds, odds and probabilities may be that the diamonds are Fair or Good). That point does not affect your actual question

For the first diamond, with a price of $326$, $x=3.95$, $y=3.98$, $z=2.43$, you can say that the model's log-odds are about $$-5.81 -0.000201 \times 326 -1.72 \times 3.95 -2.45 \times 3.98 + 7.99 \times 2.43 \approx -3$$ and the model's odds for this diamond are about $$0.00299 \times 1.000^{326} \times 0.179^{3.95} \times 0.0867^{3.98} \times 2959^{2.43} \approx 0.05$$ which is not much of a surprise as $e^{-3}\approx 0.05$

But there is no equivalent simple interpretation on how to use your column labelled probs_ratio to combine with the known values for this diamond to give a model probability also about $0.05$

$\endgroup$
  • $\begingroup$ Thanks for response here. I was looking at a similar post over on SO: stackoverflow.com/questions/41384075/… Particularly this quote "The odds ratio for the value of the intercept is the odds of a "success" when x = 0. The odds ratio for your coefficient is the increase in odds above this value of the intercept when you add one whole x value (i.e. x=1; one thought)". So, applying this to my example, does that mean that e.g. increasing feature x by 1 would result in a increase in odds of (0.179 * -1.72 )or (0.179 * -5.81)? $\endgroup$ – Doug Fir Feb 11 at 2:46
  • $\begingroup$ What is 0.05 here then. The odds. If it's not the probability of success (1) then is it 0.05:1 odds? $\endgroup$ – Doug Fir Feb 11 at 3:05
  • 1
    $\begingroup$ @DougFir Without rounding and using the coefficients I calculated from fit1, for the first diamond I would get: model log-odds of $2.986595$, model odds of $\exp(2.986595)=19.81809$ and a model probability of $\frac{19.81809}{1+19.81809}=0.9519648$. Reversing the signs of the coefficients, I would get: model log-odds of $-2.986595$, model odds of $\exp(-2.986595)=0.05045896$ and a model probability of $\frac{0.05045896}{1+0.05045896}=0.04803515$ $\endgroup$ – Henry Feb 11 at 7:54
  • 1
    $\begingroup$ @DougFir In this formulation, increasing $x$ by $1$ without changing the price or other dimensions (e.g. from $3.95$ to $4.95$ for the first diamond) would multiply the odds by $0.179$ (so changing the $\cdots \times 0.179^{3.95} \times\cdots $ term to $\cdots \times 0.179^{4.95} \times\cdots $ for the first diamond) $\endgroup$ – Henry Feb 11 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.