What is meant when we say we have a saturated model?
$\begingroup$
$\endgroup$
1
asked Jul 20, 2010 at 12:09
-
$\begingroup$ What are the relationships between the divide of saturated/non-saturated models and the divide of parametric/non-parametric models? Since saturated models amount to a trivial description of the data, are they (in spirit, at least) closer to non-parametric models than parametric models? Since non-saturated models amount to taking some form of "parametric restriction", are they (in spirit, at least) closer to parametric models than non-parametric models? $\endgroup$– Lei HuangAug 27, 2021 at 17:02
Add a comment
|
7 Answers
$\begingroup$
$\endgroup$
4
A saturated model is one in which there are as many estimated parameters as data points. By definition, this will lead to a perfect fit, but will be of little use statistically, as you have no data left to estimate variance.
For example, if you have 6 data points and fit a 5th-order polynomial to the data, you would have a saturated model (one parameter for each of the 5 powers of your independant variable plus one for the constant term).
-
19$\begingroup$ I seen examples where a model has ten data points and nine parameters. On pointing out that the model has too many parameters, I was told that the R^2 was 0.999 so the model must be correct! $\endgroup$ Jul 20, 2010 at 16:03
-
5$\begingroup$ As can be read in my and dave's post, saturated models do not per definition lead to perfect fit. but if you use the n-1 polynominal as the model they will. see Sue Doe Nihm's seminal paper on this topic psych.fullerton.edu/mbirnbaum/papers/Nihm_18_1976.pdf $\endgroup$– HenrikJul 27, 2010 at 16:52
-
$\begingroup$ Sorry if this is OT: what is the name of the case where we have an ordered collection of data points and there is a data point beyond which all cases are successes or all cases are fails? $\endgroup$– MSISMay 25, 2020 at 23:09
-
$\begingroup$
$\endgroup$
1
A saturated model is a model that is overparameterized to the point that it is basically just interpolating the data. In some settings, such as image compression and reconstruction, this isn't necessarily a bad thing, but if you're trying to build a predictive model it's very problematic.
In short, saturated models lead to extremely high-variance predictors that are being pushed around by the noise more than the actual data.
As a thought experiment, imagine you've got a saturated model, and there is noise in the data, then imagine fitting the model a few hundred times, each time with a different realization of the noise, and then predicting a new point. You're likely to get radically different results each time, both for your fit and your prediction (and polynomial models are especially egregious in this regard); in other words the variance of the fit and the predictor are extremely high.
By contrast a model that is not saturated will (if constructed reasonably) give fits that are more consistent with each other even under different noise realization, and the variance of the predictor will also be reduced.
$\begingroup$
$\endgroup$
As everybody else said before, it means that you have as much parameters have you have data points. So, no goodness of fit testing. But this does not mean that "by definition", the model can perfectly fit any data point. I can tell you by personal experience of working with some saturated models that could not predict specific data points. It is quite rare, but possible.
Another important issue is that saturated does not mean useless. For instance, in mathematical models of human cognition, model parameters are associated with specific cognitive processes that have a theoretical background. If a model is saturated, you can test its adequacy by doing focused experiments with manipulations that should affect only specific parameters. If the theoretical predictions match the observed differences (or lack of) in parameter estimates, then one can say that the model is valid.
An example: Imagine for instance a model that has two sets of parameters, one for cognitive processing, and another for motor responses. Imagine now that you have an experiment with two conditions, one in which the participants ability to respond is impaired (they can only use one hand instead of two), and in the other condition there is no impairment. If the model is valid, differences in parameter estimates for both conditions should only occur for the motor response parameters.
Also, be aware that even if one model is non-saturated, it might still be non-identifiable, which means that different combinations of parameter values produce the same result, which compromises any model fit.
If you wanna find more information on these issues in general, you might wanna take look at these papers:
Bamber, D., & van Santen, J. P. H. (1985). How many parameters can a model have and still be testable? Journal of Mathematical Psychology, 29, 443-473.
Bamber, D., & van Santen, J. P. H. (2000). How to Assess a Model's Testability and Identifiability. Journal of Mathematical Psychology, 44, 20-40.
cheers
answered Jul 27, 2010 at 13:09
$\begingroup$
$\endgroup$
1
A model is saturated if and only if it has as many parameters as it has data points (observations). Or put otherwise, in non-saturated models the degrees of freedom are bigger than zero.
This basically means that this model is useless, because it does not describe the data more parsimoniously than the raw data does (and describing data parsimoniously is generally the idea behind using a model). Furthermore, saturated models can (but don't necessarily) provide a (useless) perfect fit because they just interpolate or iterate the data.
Take for example the mean as a model for some data. If you have only one data point (e.g., 5) using the mean (i.e., 5; note that the mean is a saturated model for only one data point) does not help at all. However if you already have two data points (e.g., 5 and 7) using the mean (i.e., 6) as a model provides you with a more parsimonious description than the original data.
-
10$\begingroup$ This point about saturated not implying a perfect fit is the most interesting part of this thread. A natural example of such a situation would be monotonic regression. Suppose, e.g., you know your values must increase over time and you do polynomial regression, constraining the polynomials to be increasing. Consider data that have some error, so at some times they decrease a little. Then no matter how many parameters you use (even when it's more than the number of data values), you will never fit these data perfectly. $\endgroup$– whuber ♦Sep 10, 2011 at 16:11
$\begingroup$
$\endgroup$
1
In regression, a common use of the term "saturated model" is as follows. A saturated model has as many independent variables as there are unique levels (combinations) of the covariates. Of course this is only possible with categorical covariates. So if you have two dummy variables X1 and X2, a regression is saturated if the independent variables you include are X1, X2, and X1*X2.
This is advantageous because the conditional expectation function of Y given X1 and X2 is necessarily linear in parameters when the model is saturated (it is linear in X1, X2, X1*X2). Importantly, this model does not generally have "as many estimated parameters as data points," nor does it generally have a "perfect fit."
Here is one source for this, there are many others: "When would we expect the CEF to be linear? Two cases. One is if the data (the outcome and covariates) are multivariate Normal. The other is if the linear regression is saturated. A saturated regression model is one in which there is a parameter for each unique combination of the covariates. In this case, the regression fits the CEF perfectly because the CEF is a linear function of the dummy categories." Prof. Blackwell's lecture notes, page 2.
-
1$\begingroup$ The characterization in the first paragraph is just a special case of the quotation in the third (which is an excellent answer in its own right). $\endgroup$– whuber ♦Dec 5, 2019 at 15:27
$\begingroup$
$\endgroup$
It is also useful if you need to calculate AIC for a quasi-likelihood model. The estimate of dispersion should come from the saturated model. You would divide the LL you are fitting by the estimated dispersion from the saturated model in the AIC calculation.
$\begingroup$
$\endgroup$
In the context of SEM (or path analysis), a saturated model or a just-identified model is a model in which the number of free parameters exactly equals the number of variances and unique covariances. For example the following model is a saturated model because there are 3*4/2 data points (variances and unique covariances) and also 6 free parameters to be estimated:
