9
$\begingroup$

I am trying to make sense of what a saturated model is. AFAIK it's when you have as many features as observations.

Can we say a saturated model is a special-case of an extremely overfitted model?

$\endgroup$
  • 1
    $\begingroup$ Not exactly -- I think a saturated model used up all its degrees of freedom. It depends on the model what this means exactly. In a log-linear model, for example including all interactions in the model makes it saturated, as df=0, then, but it is not overfitted. $\endgroup$ – tomka Dec 2 '13 at 21:47
  • $\begingroup$ This thead has some good discussion of this: stats.stackexchange.com/questions/283/what-is-a-saturated-model $\endgroup$ – D L Dahly Dec 3 '13 at 13:31
6
$\begingroup$

@Tomka's right. A saturated model fits as many parameters as possible for a given set of predictors, but whether it's over-fitted or not depends on the number of observations for each unique pattern of predictors. Suppose you have a linear model with 100 observations of $y$ on $x=0$ and 100 on $x=1$. Then the model $\operatorname{E}Y = \beta_0 +\beta_1 x$ is saturated but surely not over-fitted. But if you have one observation of $y$ for each of $x=(0,1,2,3,4)^\mathrm{T}$ the model $\operatorname{E}Y = \beta_0 +\beta_1 x +\beta_2 x^2 +\beta_3 x^3 +\beta_4 x^4$ is saturated & a perfect fit—doubtless over-fitted.

When people talk about saturated models having as many parameters as observations, as in the linked web page & CV post, they're assuming a context of one observation for each predictor pattern. (Or perhaps sometimes using 'observation' differently—are 100 individuals in a 2×2 contingency table 100 observations of individuals, or 4 observations of cell frequencies?)

† Don't take "surely" & "doubtless" literally, by the way. It's possible for the first model that $\beta_1$ is so small compared to $\operatorname{Var}Y$ you'd predict better without trying to estimate it, & vice versa for the second.

$\endgroup$
  • $\begingroup$ Nice example the mapping of x={0,1} to 100 ys, thank you. Would you say this definition not accurate then: stats.gla.ac.uk/glossary/?q=node/448 ? $\endgroup$ – Ricardo Cruz Dec 3 '13 at 17:24
  • $\begingroup$ I'd say just what I said in my 2nd paragraph - it's assuming that context, & a more generally applicable definition might be better. $\endgroup$ – Scortchi Jan 11 '14 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.