3
$\begingroup$

I have a have a saturated GLM for a Bernoulli response.

Let $Y_i \backsim \text{ Ber}(\pi_i)$. The saturated model yields $\pi_i^{(s)}=y_i$, where $y_i$ is the observed value, which implies that

$$\theta_i^{(s)}=\log\!\bigg(\frac{y_i}{1-y_i}\bigg)$$

However, since $y_i=0 \text{ or } y_i=1$ isn't this problematic with the log?

Edit: It seems I've phrased myself quite poorly so I'll just say what I'm given about those models:

Saturated model - unlinks the $θ_i$

$\cdot$ Were bound by the constraint $g(\mu_i) = x_i^T\beta$ on the means

$\cdot$ Here we have one parameter $θ_i$ for each response $y_i$

$\cdot$ MLE $θ^{(s)}_i$ for $θ_i$ in the saturated model is just $θ^{(s)}_i = \arg \max_θf(y_i|θ_i)$

In particular, if we use this information for $Y_i$ which has a $\text{Bin}(m_i, \pi_i)$ distribution, we get that that

$$θ^{(s)}_i=\log(\frac{y_i}{m_i-y_i})$$

In particular, taking $m_i=1$ (making it Bernoulli), we get that

$$θ^{(s)}_i=\log(\frac{y_i}{1-y_i})$$

My issue is that $y_i=0 \text{ or } 1$, since $Y_i$ is Bernoulli, but then $θ^{(s)}_i$ seems to be undefined

I also don't understand what "were bound by the constraint..." means

$\endgroup$
3
  • $\begingroup$ What is the exponent $(s)$? What is $\theta$? Where do you get these formulae? $\endgroup$
    – AdamO
    Apr 24, 2018 at 12:18
  • 1
    $\begingroup$ You do not have to use the logit link... In your case of the saturated model you are using a non-parametric fit, with the success probability equal to the outcome. $\endgroup$
    – Knarpie
    Apr 24, 2018 at 12:37
  • 1
    $\begingroup$ @AdamO, I don't think "$(s)$" is an exponent in this case. It's just another index (somewhat analogous to "$_i$") stating that this parameter is from the model denoted $s$. Presumably, $s$ is for "saturated". Furthermore, I suspect "$\theta$" refers generically to the set of parameters from that model, specifically with $\theta_i = \pi_i$. $\endgroup$ Apr 25, 2018 at 13:54

2 Answers 2

2
$\begingroup$

A saturated model means that you are fitting a separate parameter for every data point. In general, the Bernoulli distribution has a parameter, $\pi$, that ranges over the interval $(0, 1)$. That is, the parameter cannot really equal $0$ or $1$, only asymptotically approach it. When you fit a separate parameter for each datum, as you do here, you fit either $\pi_i = 0$ or $\pi_i=1$. This means you are asserting there is no possible variation for that observation. You are no longer really fitting a Bernoulli, but instead a degenerate distribution. The logit is not an appropriate link function for the degenerate distribution. That's the issue.

This isn't really a 'good' model, so I wouldn't worry too much about this. The point of a saturated model is typically to provide an extreme case that can be used in a comparison. For example, you could test a fitted model against the saturated model to see if your model seems to be leaving something out (although you wouldn't necessarily know what). It doesn't really matter that the saturated model is unlikely to be a good model for other purposes. You would use it for that test and then set it aside.

$\endgroup$
4
  • $\begingroup$ Is that why we don't usually want to compare the deviance of a bernoulli GLM to a $\chi^2$ distribution? Or is it just because it depends only on the data? $\endgroup$
    – asdf
    Apr 25, 2018 at 14:07
  • $\begingroup$ I'm not sure I follow that, @asdf. To test a fitted model against the saturated model, we usually just test the deviance against $0$, w/o actually having to bother w/ 'fitting' a saturated model. Such a test should have a $\chi^2$ distribution with the degrees of freedom equal to the difference between the number of parameters. $\endgroup$ Apr 25, 2018 at 14:15
  • $\begingroup$ Sorry, I was talking about model selection - the value which we are interested in is the deviance and we test by comparing it to a $\chi^2$ distribution. But the lecture notes say that this is not a good test for a bernoulli GLMs $\endgroup$
    – asdf
    Apr 25, 2018 at 14:19
  • $\begingroup$ Off the top of my head, I don't really know what they are referring to, @asdf. $\endgroup$ Apr 25, 2018 at 14:27
1
$\begingroup$

That is not the way I interpret a saturated model. The saturated model is applicable to a regression model where the unique exposure levels have, as predicted values, the empirical frequency of outcomes for every level. That means, if $X$ takes values 1, 2, and 3, the predicted value of $Y$ at each of those levels (from the regression model) is $E[Y| X=1]$, $E[Y| X=2]$, and $E[Y| X=3]$. Hopefully there is a sufficient sample size so that the marginal frequencies of $X$ do not have small cell counts.

If there is only one observation per exposure level, you are correct that the predicted value is just the $Y$ value: 0 or 1. Characterizing those predictions with log odds ratios requires a value of $-\infty$ or $\infty$ which is the limit of $\log(p/(1-p))$ as $p \rightarrow 0$ or $1$ respectively. So your expression is correct. The solution to the MLE is said to lie on the boundary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.