2
$\begingroup$

I have a have a saturated GLM for a Bernoulli response.

Let $Y_i \backsim \text{ Ber}(\pi_i)$. The saturated model yields $\pi_i^{(s)}=y_i$, where $y_i$ is the observed value, which implies that

$$\theta_i^{(s)}=\log\!\bigg(\frac{y_i}{1-y_i}\bigg)$$

However, since $y_i=0 \text{ or } y_i=1$ isn't this problematic with the log?

Edit: It seems I've phrased myself quite poorly so I'll just say what I'm given about those models:

Saturated model - unlinks the $θ_i$

$\cdot$ Were bound by the constraint $g(\mu_i) = x_i^T\beta$ on the means

$\cdot$ Here we have one parameter $θ_i$ for each response $y_i$

$\cdot$ MLE $θ^{(s)}_i$ for $θ_i$ in the saturated model is just $θ^{(s)}_i = \arg \max_θf(y_i|θ_i)$

In particular, if we use this information for $Y_i$ which has a $\text{Bin}(m_i, \pi_i)$ distribution, we get that that

$$θ^{(s)}_i=\log(\frac{y_i}{m_i-y_i})$$

In particular, taking $m_i=1$ (making it Bernoulli), we get that

$$θ^{(s)}_i=\log(\frac{y_i}{1-y_i})$$

My issue is that $y_i=0 \text{ or } 1$, since $Y_i$ is Bernoulli, but then $θ^{(s)}_i$ seems to be undefined

I also don't understand what "were bound by the constraint..." means

$\endgroup$
  • $\begingroup$ What is the exponent $(s)$? What is $\theta$? Where do you get these formulae? $\endgroup$ – AdamO Apr 24 '18 at 12:18
  • 1
    $\begingroup$ You do not have to use the logit link... In your case of the saturated model you are using a non-parametric fit, with the success probability equal to the outcome. $\endgroup$ – Knarpie Apr 24 '18 at 12:37
  • 1
    $\begingroup$ @AdamO, I don't think "$(s)$" is an exponent in this case. It's just another index (somewhat analogous to "$_i$") stating that this parameter is from the model denoted $s$. Presumably, $s$ is for "saturated". Furthermore, I suspect "$\theta$" refers generically to the set of parameters from that model, specifically with $\theta_i = \pi_i$. $\endgroup$ – gung - Reinstate Monica Apr 25 '18 at 13:54
2
$\begingroup$

A saturated model means that you are fitting a separate parameter for every data point. In general, the Bernoulli distribution has a parameter, $\pi$, that ranges over the interval $(0, 1)$. That is, the parameter cannot really equal $0$ or $1$, only asymptotically approach it. When you fit a separate parameter for each datum, as you do here, you fit either $\pi_i = 0$ or $\pi_i=1$. This means you are asserting there is no possible variation for that observation. You are no longer really fitting a Bernoulli, but instead a degenerate distribution. The logit is not an appropriate link function for the degenerate distribution. That's the issue.

This isn't really a 'good' model, so I wouldn't worry too much about this. The point of a saturated model is typically to provide an extreme case that can be used in a comparison. For example, you could test a fitted model against the saturated model to see if your model seems to be leaving something out (although you wouldn't necessarily know what). It doesn't really matter that the saturated model is unlikely to be a good model for other purposes. You would use it for that test and then set it aside.

$\endgroup$
  • $\begingroup$ Is that why we don't usually want to compare the deviance of a bernoulli GLM to a $\chi^2$ distribution? Or is it just because it depends only on the data? $\endgroup$ – asdf Apr 25 '18 at 14:07
  • $\begingroup$ I'm not sure I follow that, @asdf. To test a fitted model against the saturated model, we usually just test the deviance against $0$, w/o actually having to bother w/ 'fitting' a saturated model. Such a test should have a $\chi^2$ distribution with the degrees of freedom equal to the difference between the number of parameters. $\endgroup$ – gung - Reinstate Monica Apr 25 '18 at 14:15
  • $\begingroup$ Sorry, I was talking about model selection - the value which we are interested in is the deviance and we test by comparing it to a $\chi^2$ distribution. But the lecture notes say that this is not a good test for a bernoulli GLMs $\endgroup$ – asdf Apr 25 '18 at 14:19
  • $\begingroup$ Off the top of my head, I don't really know what they are referring to, @asdf. $\endgroup$ – gung - Reinstate Monica Apr 25 '18 at 14:27
1
$\begingroup$

That is not the way I interpret a saturated model. The saturated model is applicable to a regression model where the unique exposure levels have, as predicted values, the empirical frequency of outcomes for every level. That means, if $X$ takes values 1, 2, and 3, the predicted value of $Y$ at each of those levels (from the regression model) is $E[Y| X=1]$, $E[Y| X=2]$, and $E[Y| X=3]$. Hopefully there is a sufficient sample size so that the marginal frequencies of $X$ do not have small cell counts.

If there is only one observation per exposure level, you are correct that the predicted value is just the $Y$ value: 0 or 1. Characterizing those predictions with log odds ratios requires a value of $-\infty$ or $\infty$ which is the limit of $\log(p/(1-p))$ as $p \rightarrow 0$ or $1$ respectively. So your expression is correct. The solution to the MLE is said to lie on the boundary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.