Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I just read about the deviance measure for the logistic regression. However, the part that is called saturated model is not clear to me.

I did an extensive Google search but none of the results answered my question. So far I found out that a saturated model has a parameter for each observation which as a consequence results in a perfect fit. This is clear to me. But: further the fitted values (of a saturated model) are equal to the observed values.

Since from my knowledge, logistic regression is used for classification the given observed data are covariates with additional labels $y \in \{0,1\}$. However, the deviance measure employs probabilities but not the actual labels. One applies the calculated predicted probability of the logistic regression versus observed probabilities. However, since one has only given labels instead of probabilities I'm confused how to construct a saturated model from these labels?

share|improve this question

1 Answer 1

up vote 5 down vote accepted

For each $y_i$, the fitted probability from the saturated model will be the same as $y_i$, either zero or one. As explained here, the likelihood of the saturated model is $1$. Therefore, the deviance of such model will be $-2\log(1/1) = 0$, on $0$ df. Here is an example from R:

y = c(1,1,1,0,0,0)
a <- factor(1:length(y)) 
fit <- glm(y~a,family=binomial) 
summary(fit)

Deviance Residuals: 
 0  0  0  0  0  0

Null deviance: 8.3178e+00  on 5  degrees of freedom

Residual deviance: 2.5720e-10  on 0  degrees of freedom

The saturated model always has $n$ parameters where $n$ is the sample size. That's why the null deviance is always on $(n - 1)$ df, since the null model has only the intercept. E.g., if I add one replicate for each of the six factor levels, I will get the following:

> k2
 [1] 1 2 3 4 5 6 1 2 3 4 5 6
Levels: 1 2 3 4 5 6
> y2
 [1] 1 1 1 0 0 0 1 1 1 0 0 0
> fit3 = glm(y2 ~ k2, family = binomial)
> summary(fit3)    

    Null deviance: 1.6636e+01  on 11  degrees of freedom
    Residual deviance: 5.1440e-10  on  6  degrees of freedom
share|improve this answer
    
Are you sure? If $y_i$ was $0$ the denominator in the deviance expression becomes $0$ as well which is then not defined. –  toom Sep 2 at 19:07
    
Ah, yeah I see. Thanks for your answer. :) –  toom Sep 2 at 19:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.