# Iterative proportional fitting and independent variables

I'm trying to understand the "classic" iterative proportional fitting (IPF) algorithm.

Does it always assume that the variables being analyzed are independent? If the variables are independent, then can't we just compute each joint count from the product of two marginal counts?

For example, in the wikipedia example referenced above, are the two variables (gender and handedness) independent? The final joint counts are

45.24 = 100 * 52/100 * 87/100
41.76 = 100 * 48/100 * 87/100
6.76 = 100 * 52/100 * 13/100
6.24 = 100 * 48/100 * 13/100


so no iterative algorithm is necessary.

• Are you still trying to understand the classic IPF algorithm? Did any of the three answers help? – conjugateprior Oct 23 '12 at 9:36
• @ConjugatePrior: I understand things much better now. Thanks for your excellent answer! – Tyler Streeter Oct 25 '12 at 19:53

What you are seeing in the example is just a function of looking at a simple two-way table where there aren't any particularly interesting log linear models between independence and saturation.

In a three way table things may be a bit clearer. Take for example R's built-in data set HairEyeColor. This has dimensions Hair, Eye and Sex.

ftable(addmargins(HairEyeColor)) ## the data with all margins

Sex Male Female Sum
Hair  Eye
Black Brown       32     36  68
Blue        11      9  20
Hazel       10      5  15
Green        3      2   5
Sum         56     52 108
Brown Brown       53     66 119
Blue        50     34  84
Hazel       25     29  54
Green       15     14  29
Sum        143    143 286
Red   Brown       10     16  26
Blue        10      7  17
Hazel        7      7  14
Green        7      7  14
Sum         34     37  71
Blond Brown        3      4   7
Blue        30     64  94
Hazel        5      5  10
Green        8      8  16
Sum         46     81 127
Sum   Brown       98    122 220
Blue       101    114 215
Hazel       47     46  93
Green       33     31  64
Sum        279    313 592


There are quite a few log linear models we might consider, all of which can be fitted by IPF and some of which need to be (or at least need an iterative procedure of some form). R's loglin function does fit in fact fit them all with IPF, whether or not a closed form is available. Let's consider two models.

IPF estimates cell values subject to marginal constraints. This corresponds to log linear models containing (or rather lacking) particular interaction terms. For example, suppose you got the idea that the connection between hair and eye colour was due entirely to (and thus conditionally independent given) sex. You'd then fit a model that contained a grand mean, main effects for Sex, Eye, and Hair, and two interaction terms Hair-Sex and Eye-Sex. This model can be written $[Hair,Sex][Eye,Sex]$ (or as the undirected graphical model $Hair-Sex-Eye$).

If you were to add Eye-Hair and Eye-Hair-Sex terms in the model then it would be saturated. If you were to take out the interaction terms then you'd be back to the independence model $[Hair][Sex][Eye]$.

Here's the conditional independence model in R:

mod1 <- loglin(HairEyeColor, list(c('Eye', 'Sex'), c('Hair', 'Sex')),
fit=TRUE)


Looking at its fitted values shows which margins are recovered and which not:

ftable(addmargins(mod1$fit)) Sex Male Female Sum Hair Eye Black Brown 19.670251 20.268371 39.938622 Blue 20.272401 18.939297 39.211699 Hazel 9.433692 7.642173 17.075864 Green 6.623656 5.150160 11.773816 Sum 56.000000 52.000000 108.000000 Brown Brown 50.229391 55.738019 105.967410 Blue 51.767025 52.083067 103.850092 Hazel 24.089606 21.015974 45.105580 Green 16.913978 14.162939 31.076918 Sum 143.000000 143.000000 286.000000 Red Brown 11.942652 14.421725 26.364378 Blue 12.308244 13.476038 25.784282 Hazel 5.727599 5.437700 11.165298 Green 4.021505 3.664537 7.686042 Sum 34.000000 37.000000 71.000000 Blond Brown 16.157706 31.571885 47.729591 Blue 16.652330 29.501597 46.153927 Hazel 7.749104 11.904153 19.653257 Green 5.440860 8.022364 13.463224 Sum 46.000000 81.000000 127.000000 Sum Brown 98.000000 122.000000 220.000000 Blue 101.000000 114.000000 215.000000 Hazel 47.000000 46.000000 93.000000 Green 33.000000 31.000000 64.000000 Sum 279.000000 313.000000 592.000000  Notice that all the margins except the one in the right hand column are the same as in the original data. An alternative model that captures this margin while still being neither the pure independence model nor saturated is$[Hair,Sex][Eye,Sex][Eye,Hair]$: mod2 <- loglin(HairEyeColor, list(c('Eye', 'Sex'), c('Hair', 'Sex'), c('Hair','Eye')), fit=TRUE)  This model assumes that all odds ratios between any two variables are the same at each level of the third ('homogeneous association'). This takes six IPF iterations to fit and has no closed form estimates. ftable(addmargins(mod2$fit))

Sex       Male     Female        Sum
Hair  Eye
Black Brown      32.793454  35.206546  68.000000
Blue       11.743353   8.256647  20.000000
Hazel       8.444542   6.555458  15.000000
Green       3.018490   1.981510   5.000000
Sum        55.999839  52.000161 108.000000
Brown Brown      52.524536  66.475464 119.000000
Blue       45.930572  38.069428  84.000000
Hazel      28.196301  25.803699  54.000000
Green      16.348411  12.651589  29.000000
Sum       142.999820 143.000180 286.000000
Red   Brown      10.760594  15.239406  26.000000
Blue        8.819781   8.180219  17.000000
Hazel       6.916815   7.083185  14.000000
Green       7.502806   6.497194  14.000000
Sum        33.999996  37.000004  71.000000
Blond Brown       1.926579   5.073421   7.000000
Blue       34.500428  59.499572  94.000000
Hazel       3.443330   6.556670  10.000000
Green       6.129759   9.870241  16.000000
Sum        46.000097  80.999903 127.000000
Sum   Brown      98.005164 121.994836 220.000000
Blue      100.994134 114.005866 215.000000
Hazel      47.000988  45.999012  93.000000
Green      32.999465  31.000535  64.000000
Sum       278.999751 313.000249 592.000000


As expected, its fitted values now also recover the third column margins that mod1 did not because of the extra interaction term.

As @gjabel notes, Agresti 2002 contains a nice clear discussion. Hope that helps.

• Thanks for walking through an example... the 3-way example really helps clarify things. – Tyler Streeter Oct 23 '12 at 19:22

A variation of the IPF is frequently used in survey statistics to adjust for the error in the data, where it is known as the raking ratio adjustment. Initially it was proposed to deal with sampling error (the difference between the true proportions and the observed proportions due to the fact that not everybody is being observed), and these days it is used more to correct for non-sampling error (not all values are observed because of non-response). Starting from the sample values, the weights are adjusted so that a given margin corresponds to the known population value, until convergence. This algorithm solves a certain optimization problem of finding the joint distribution that is closest to that of the sample, yet satisfies the marginal distributions. See http://www.citeulike.org/user/ctacmo/article/10383730/.

Lacking any additional information about the joint distribution, the joint independence is, in a way, the most natural assumption you could possibly make.

You can use IPF to fit any log-linear models. If your model is relatively simple and has no interactions terms, then no iterations are required. For example the model being fitted in the wikipedia article is: $$\log m_{ij} = \beta_i + \beta_j .$$ However, if you are fitting a more complex model, perhaps with interactions over contingency tables with more than two dimensions then you will need to to iterate to find the mle. Agresti's chapter on log-linear models (in his Categorical Data Analysis book) explains which log-linear models can be solved directly, and which need an iterative solution.