# Saturated model for a Bernoulli response

I have a have a saturated GLM for a Bernoulli response.

Let $Y_i \backsim \text{ Ber}(\pi_i)$. The saturated model yields $\pi_i^{(s)}=y_i$, where $y_i$ is the observed value, which implies that

$$\theta_i^{(s)}=\log\!\bigg(\frac{y_i}{1-y_i}\bigg)$$

However, since $y_i=0 \text{ or } y_i=1$ isn't this problematic with the log?

Edit: It seems I've phrased myself quite poorly so I'll just say what I'm given about those models:

Saturated model - unlinks the $θ_i$

$\cdot$ Were bound by the constraint $g(\mu_i) = x_i^T\beta$ on the means

$\cdot$ Here we have one parameter $θ_i$ for each response $y_i$

$\cdot$ MLE $θ^{(s)}_i$ for $θ_i$ in the saturated model is just $θ^{(s)}_i = \arg \max_θf(y_i|θ_i)$

In particular, if we use this information for $Y_i$ which has a $\text{Bin}(m_i, \pi_i)$ distribution, we get that that

$$θ^{(s)}_i=\log(\frac{y_i}{m_i-y_i})$$

In particular, taking $m_i=1$ (making it Bernoulli), we get that

$$θ^{(s)}_i=\log(\frac{y_i}{1-y_i})$$

My issue is that $y_i=0 \text{ or } 1$, since $Y_i$ is Bernoulli, but then $θ^{(s)}_i$ seems to be undefined

I also don't understand what "were bound by the constraint..." means

• What is the exponent $(s)$? What is $\theta$? Where do you get these formulae? – AdamO Apr 24 '18 at 12:18
• You do not have to use the logit link... In your case of the saturated model you are using a non-parametric fit, with the success probability equal to the outcome. – Knarpie Apr 24 '18 at 12:37
• @AdamO, I don't think "$(s)$" is an exponent in this case. It's just another index (somewhat analogous to "$_i$") stating that this parameter is from the model denoted $s$. Presumably, $s$ is for "saturated". Furthermore, I suspect "$\theta$" refers generically to the set of parameters from that model, specifically with $\theta_i = \pi_i$. – gung - Reinstate Monica Apr 25 '18 at 13:54

A saturated model means that you are fitting a separate parameter for every data point. In general, the Bernoulli distribution has a parameter, $\pi$, that ranges over the interval $(0, 1)$. That is, the parameter cannot really equal $0$ or $1$, only asymptotically approach it. When you fit a separate parameter for each datum, as you do here, you fit either $\pi_i = 0$ or $\pi_i=1$. This means you are asserting there is no possible variation for that observation. You are no longer really fitting a Bernoulli, but instead a degenerate distribution. The logit is not an appropriate link function for the degenerate distribution. That's the issue.
• Is that why we don't usually want to compare the deviance of a bernoulli GLM to a $\chi^2$ distribution? Or is it just because it depends only on the data? – asdf Apr 25 '18 at 14:07
• I'm not sure I follow that, @asdf. To test a fitted model against the saturated model, we usually just test the deviance against $0$, w/o actually having to bother w/ 'fitting' a saturated model. Such a test should have a $\chi^2$ distribution with the degrees of freedom equal to the difference between the number of parameters. – gung - Reinstate Monica Apr 25 '18 at 14:15
• Sorry, I was talking about model selection - the value which we are interested in is the deviance and we test by comparing it to a $\chi^2$ distribution. But the lecture notes say that this is not a good test for a bernoulli GLMs – asdf Apr 25 '18 at 14:19
That is not the way I interpret a saturated model. The saturated model is applicable to a regression model where the unique exposure levels have, as predicted values, the empirical frequency of outcomes for every level. That means, if $X$ takes values 1, 2, and 3, the predicted value of $Y$ at each of those levels (from the regression model) is $E[Y| X=1]$, $E[Y| X=2]$, and $E[Y| X=3]$. Hopefully there is a sufficient sample size so that the marginal frequencies of $X$ do not have small cell counts.
If there is only one observation per exposure level, you are correct that the predicted value is just the $Y$ value: 0 or 1. Characterizing those predictions with log odds ratios requires a value of $-\infty$ or $\infty$ which is the limit of $\log(p/(1-p))$ as $p \rightarrow 0$ or $1$ respectively. So your expression is correct. The solution to the MLE is said to lie on the boundary.