I can't understand why is saturated model perfectly fitted? I know the definition, I just don't have any intuition.

  • $\begingroup$ See here. $\endgroup$ – Scortchi Jan 31 '15 at 14:19
  • 1
    $\begingroup$ How much lack of fit do you expect when you draw a straight line through two points, or lay a tabletop on three legs? $\endgroup$ – Glen_b Feb 1 '15 at 3:11

One way to get some intuition for staturation is to look carefully at parameter indices. I'll use an example from contingency table analysis, because I think it's a little bit clearer than the regression context (and also because regression versions have already been pointed to).

Consider the log linear models of a two way contingency table $C$. Notationally, the $i$th row and $j$th column value is denoted $C_{ij}$. Textbooks define a sequence of models for the log of the expected or mean value of the these entries $E[C_{ij}]$, in the same way that regression models build a model of the expected value of some $E[Y_i]$ in terms of some function of covariates.

Assuming you have some intuition for the idea of link functions (here the log) and the idea of modeling the expected value of something rather than its actual value, then the intuition for saturation comes like this:

The simplest model of $C$ says that $$ \log E[C_{ij}] = \lambda $$ this implies that all entries should, according to the model, have the same expected value because the parameter $\lambda$ has no index, meaning it applies to all $i$ and all $j$. (Simplifying slightly, parameters are distinct only when they have an index that can vary, and they can have as many possible distinct values as there are possible values of that index.)

The next two most complex models $$ \log E[C_{ij}] = \lambda + \lambda_i $$ and $$ \log E[C_{ij}] = \lambda + \lambda_j $$ say that each row (column) has the same expected value but different rows (columns) can have different expected values, because the new parameters are indexed by $i$ but not $j$ ($j$ but not $i$) These models allow as many different expected cell values as there are rows (columns) but that's still fewer distinct values than there are cell entries in total, so the model is not saturated and the expected values won't typically coincide exactly with the actual values.

Finally the most complex model in this sequence is $$ \log E[C_{ij}] = \lambda + \lambda_j + \lambda_j + \lambda_{ij} $$ The indexing indicates that there can be as many distinct expected values as there are entries in the table. Entries in $C$ are labelled by all combinations of $i$ and $j$ and the model has parameters corresponding to all combinations of $i$ and $j$ too, so every cell can effectively get its own parameter. When that is the case (and when we fit by Maximum Likelihood) the expected cell value coincide with the actual cell values.

In more detail, and comparing with the second model: the expected cell values in row $i$ are forced to be all the same, even though the actual cell values in row $i$ probably aren't, so we can interpret the difference between the two as unmodelled variation from a mean. In the final model there is no variation around the model's expected values for each cell, because there are no other cells with the same mean to compare it with, so the expected cell value are also the actual values.

Addendum: There is a completely parallel development of these ideas based not on the parameters of various different types of model, but on which table sums and margins each model must respect. In this sequence, the first of our models need only agree with $C$ about what the total sum of the entries is. In the second model it must agree about the total sum and also get the same row totals as $C$. In the third model it must agree about the total sum and also get the same column totals as $C$. And in the final model it must reflect all the entry values. But if you have the intuition about the parameters then it should make sense why these margin capturing relationships hold.

  • $\begingroup$ I think that I understood the idea, thanks. But you consider only the case where independent variables are discreete (or even binary). What should I understand as saturated model when my variables are continuous? $\endgroup$ – Marta Jan 31 '15 at 15:38

Saturated essentially means "perfectly fitted". In a saturated model there are as many estimated parameters as there are observations.

Take a simple case of only 2 observations. Say "height" is the DV and one person is male and one female. Now a model with just "sex" as an IV will fit perfectly. In a linear model, the intercept would be the average height of the two people and the effect of (say) maleness with be the difference between the man and the woman.

  • 1
    $\begingroup$ But with four observations (two for each sex) the model won't fit perfectly, yet it's still saturated (unless you have another IV up your sleeve). $\endgroup$ – Scortchi Jan 31 '15 at 14:19

Looking at the answers to this thread

A saturated model is one in which there are as many estimated parameters as data points.

Consider the case of linear regression. You have a sample of size $n$ and $k$ regressors including the constant term. Then you essentially are trying to "solve" (i.e. find the $k$ unknown parameters that satisfy) a set of linear equations

$$\begin{align} y_1 = b_1 + b_2x_{21} + ...+ b_{k}x_{k1} \\ y_2 = b_1 + b_2x_{22} + ...+ b_{k}x_{k2}\\ ...\\ y_n = b_1 + b_2x_{2n} + ...+ b_{k}x_{kn}\\ \end{align}$$

In matrix notation

$$\mathbf Y_{n \times 1} = \mathbf X_{n \times k}\mathbf b_{k \times 1}$$

As long as $k<n$ the system cannot be solved exactly, because we have more equations than unknowns, and so we resort to an approximate solution, under some optimality criterion, say, a parameter vector that minimizes the sum of squared deviations. But if $k=n$ then we have as many unknowns as equations. This is reflected in the fact that now, the matrix $\mathbf X$ has become a square matrix, and we are looking at

$$\mathbf Y_{n \times 1} = \mathbf X_{n \times n}\mathbf b_{n \times 1}$$

Then from high-school algebra, we know that as long as the $\mathbf X$ matrix is invertible ("determinant not zero"), then the system has a unique solution

$$\mathbf b_{n \times 1} = \mathbf X^{-1}_{n \times n}\mathbf Y_{n \times 1}$$

But this is a solution in the mathematical, exact sense, of the calculated parameter vector satisfying exactly each and everyone of the $n$ equations - no "residuals" -which, in statistical terminology, is the equivalent of "perfect fit".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.