Logistic Regression : How to obtain a saturated model

Question

I just read about the deviance measure for the logistic regression. However, the part that is called saturated model is not clear to me.

I did an extensive Google search but none of the results answered my question. So far I found out that a saturated model has a parameter for each observation which as a consequence results in a perfect fit. This is clear to me. But: further the fitted values (of a saturated model) are equal to the observed values.

Since from my knowledge, logistic regression is used for classification the given observed data are covariates with additional labels $y \in \{0,1\}$. However, the deviance measure employs probabilities but not the actual labels. One applies the calculated predicted probability of the logistic regression versus observed probabilities. However, since one has only given labels instead of probabilities I'm confused how to construct a saturated model from these labels?

Answer 1

For each $y_i$, the fitted probability from the saturated model will be the same as $y_i$, either zero or one. As explained here, the likelihood of the saturated model is $1$. Therefore, the deviance of such model will be $-2\log(1/1) = 0$, on $0$ df. Here is an example from R:

y = c(1,1,1,0,0,0)
a <- factor(1:length(y)) 
fit <- glm(y~a,family=binomial) 
summary(fit)

Deviance Residuals: 
 0  0  0  0  0  0

Null deviance: 8.3178e+00  on 5  degrees of freedom

Residual deviance: 2.5720e-10  on 0  degrees of freedom

The saturated model always has $n$ parameters where $n$ is the sample size. That's why the null deviance is always on $(n - 1)$ df, since the null model has only the intercept. E.g., if I add one replicate for each of the six factor levels, I will get the following:

> k2
 [1] 1 2 3 4 5 6 1 2 3 4 5 6
Levels: 1 2 3 4 5 6
> y2
 [1] 1 1 1 0 0 0 1 1 1 0 0 0
> fit3 = glm(y2 ~ k2, family = binomial)
> summary(fit3)    

    Null deviance: 1.6636e+01  on 11  degrees of freedom
    Residual deviance: 5.1440e-10  on  6  degrees of freedom

Actually, it turns out that in R what the saturated model is depends on the form of input even if the data are exactly the same, which is not very nice. In particular, in the example above there are 12 observations and 6 factor levels, so the saturated model should have had 6 parameters, not 12. In general, a saturated model is defined as one where the number of parameters is equal to the number of distinct covariate patterns. I have no idea why R code "admitted" that factor k2 has 6 distinct levels, and yet the saturated model was fitted with 12 parameters.

Now, if we use exactly the same data in "binomial" form, we'll get a correct answer:

y_yes = 2 * c(1,1,1,0,0,0)
y_no = 2 * c(0,0,0,1,1,1)
x = factor(c(1:6))

> x
[1] 1 2 3 4 5 6
Levels: 1 2 3 4 5 6
> y_yes
[1] 2 2 2 0 0 0
> y_no
[1] 0 0 0 2 2 2

modelBinomialForm = glm(cbind(y_yes, y_no) ~ x, family=binomial)

Deviance Residuals: 
[1]  0  0  0  0  0  0

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)  2.490e+01  1.096e+05       0        1
x2           1.375e-08  1.550e+05       0        1
x3           1.355e-08  1.550e+05       0        1
x4          -4.980e+01  1.550e+05       0        1
x5          -4.980e+01  1.550e+05       0        1
x6          -4.980e+01  1.550e+05       0        1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1.6636e+01  on 5  degrees of freedom
Residual deviance: 3.6749e-10  on 0  degrees of freedom

Now we see that the saturated model has 6 parameters and it coincides with the fitted model. Hence, the null deviance is on (6 - 1) = 5 df, and the residual deviance is on (6-6) = 0 df.