How can a regression model be any use if you don't know the function you are trying to get the parameters for?

I saw a piece of research that said that mothers who breast fed their children were less likely to suffer diabetes in later life. The research was from a survey of some 1000 mothers and controlled for miscellaneous factors and a loglinear model was used.

Now does this mean that they reckon all the factors that determine the likelihood of diabetes fit in a nice function (exponential presumably) that translates neatly into a linear model with logs and that whether the woman breast fed turned out to be statistically significant?

I'm missing something I'm sure but, how the hell do they know the model?

  • $\begingroup$ Thank you all very much. I want to spend a little time thinking about your answers and perhaps, if you don't mind my try writing them in my terms for your views. I like this description of the process as coming from the Taylor series. I have had to pick up my knowledge of regression haphazzardly and through Economics and Mathematics for Economists and the link with Taylor is noteable by it's absense. $\endgroup$ Jan 6 '11 at 14:09
  • $\begingroup$ I have merged your accounts; but please, register it here stats.stackexchange.com/users/login so you'd not loose it again. $\endgroup$
    – user88
    Jan 6 '11 at 15:52

It helps to view regression as a linear approximation of the true form. Suppose the true relationship is


with $x_1,...,x_k$ factors explaining the $y$. Then first order Taylor approximation of $f$ around zero is:

$$f(x_1,...,x_k)=f(0,...,0)+\sum_{i=1}^{k}\frac{\partial f(0)}{\partial x_k}x_k+\varepsilon,$$

where $\varepsilon$ is the approximation error. Now denote $\alpha_0=f(0,...,0)$ and $\alpha_k=\frac{\partial{f}(0)}{\partial x_k}$ and you have a regression:

$$y=\alpha_0+\alpha_1 x_1+...+\alpha_k x_k + \varepsilon$$

So although you do not know the true relationship, if $\varepsilon$ is small you get approximation, from which you can still deduce useful conclusions.

  • 1
    $\begingroup$ Hi, very nice explanation but but I don't manage to understand the "sigma" part in the taylor series expansion. How do you reduce this equation found here: mathworld.wolfram.com/TaylorSeries.html under "A Taylor series of a real function in two variables" to yours? $\endgroup$
    – Arun
    May 11 '12 at 12:47
  • 1
    $\begingroup$ @Arun, take $n=1$ in formula (32). $\endgroup$
    – mpiktas
    May 11 '12 at 19:00

The other side of the answer, complementary to mpiktas's answer but not mentioned so far, is:

"They don't, but as soon as they assume some model structure, they can check it against the data".

The two basic things that could go wrong are: The form of the function, e.g. it's not even linear in logs. So you'd start by plotting the an appropriate residual against the expected values. Or the choice of conditional distribution, e.g. the observed counts overdispersed relative to Poisson. So you'd test against an Negative Binomial version of the same model, or see if extra covariates account for the extra variation.

You'd also want to check for outliers, influential observations, and a host of other things. A reasonable place to read about checking these kinds of model problems is ch.5 of Cameron and Trivedi 1998. (There is surely a better place for epidemiologically oriented researchers to start - perhaps other folk can suggest it.)

If these diagnostics indicated the model failed to fit the data, you'd change the relevant aspect of the model and start the whole process again.

  • 1
    $\begingroup$ +1 This is the key that keeps it all from being hand-waving: you don't know, but you try something and then look at how well it matches and in what way it mismatches your data. $\endgroup$
    – Wayne
    Jul 2 '12 at 1:37

An excellent first question! I agree with mpiktas's answer, i.e. the short answer is "they don't, but they hope to have an approximation to the right model that gives approximately the right answer".

In the jargon of epidemiology, this model uncertainty is one source of what's known as 'residual confounding'. See Steve Simon's page 'What is residual confounding?' for a good short description, or Heiko Becher's 1992 paper in Statistics in Medicine (subscription req'd) for a longer, more mathematical treatment, or Fewell, Davey Smith & Sterne's more recent paper in the American Journal of Epidemiology (subscription req'd).

This is one reason that epidemiology of small effects is difficult and the findings often controversial - if the measured effect size is small, it's hard to rule out residual confounding or other sources of bias as the explanation.

  • 1
    $\begingroup$ I'd argue that model misspecification - which seems to be what the OP is talking about, is somewhat distinct from residual confounding. Confounding requires a covariate. You can screw up a regression with just the misspecification of an exposure and outcome. $\endgroup$
    – Fomite
    Jul 1 '12 at 23:47

There is the famous quote "Essentially, all models are wrong, but some are useful" of George Box. When fitting models like this, we try to (or should) think about the data generation process and the physical, real world, relationships between the response and covariates. We try to express these relationships in a model that fits the data. Or to put it another way, is consistent with the data. As such an empirical model is produced.

Whether it is useful or not is determined later - does it give good, reliable predictions, for example, for women not used to fit the model? Are the model coefficients interpretable and of scientific use? Are the effect sizes meaningful?


The answers you have already gotten are excellent ones, but I'm going to give a (hopefully) complementary answer from the perspective of an Epidemiologist. I really have three thoughts on this:

First, they don't. See also: All models are wrong, some models are useful. The goal is not to produce a single, definitive number that is taken as the "truth" of an underlying function. The goal is to produce an estimate of that function, with a quantification of the uncertainty around it, that is a reasonable and useful approximation of the underlying function.

This is especially true for large effect measures. The "take away" message from a study that finds a relative risk of 3.0 isn't really different if the "true" relationship is 2.5 or 3.2. As @onestop mentioned, this does get harder with small effect measure estimates, because the difference between 0.9, 1.0 and 1.1 can be huge from a health and policy standpoint.

Second, there's a process hidden in most Epidemiology papers. That's the actual model selection process. We tend to report the model we ended up with, not all the models we considered (because that would be tiresome, if nothing else). There are a slew of model building steps, conceptual diagrams, diagnostics, fit statistics, sensitivity analysis, swearing at computers and scribbling on white boards involved in the analysis of even small observational studies.

Because while you are making assumptions, many of them are also assumptions you can check.

Third, sometimes we don't. And then we go to conferences and argue with each other about it ;)

If you're interested in the nuts and bolts of Epidemiology as a field, and how we perform out research, the best place to start is probably Modern Epidemiology 3rd Edition by Rothman, Greenland and Lash. It's a moderately technical and very good overview of how Epi research is conducted.

  • 1
    $\begingroup$ +1, this is a good complement to what's here. It's nice to see that a useful contribution can still be made, even after so many other good ones already exist. $\endgroup$ Jul 2 '12 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.