Understanding regressions - the role of the model

Question

How can a regression model be any use if you don't know the function you are trying to get the parameters for?

I saw a piece of research that said that mothers who breast fed their children were less likely to suffer diabetes in later life. The research was from a survey of some 1000 mothers and controlled for miscellaneous factors and a loglinear model was used.

Now does this mean that they reckon all the factors that determine the likelihood of diabetes fit in a nice function (exponential presumably) that translates neatly into a linear model with logs and that whether the woman breast fed turned out to be statistically significant?

I'm missing something I'm sure but, how the hell do they know the model?

Answer 1

It helps to view regression as a linear approximation of the true form. Suppose the true relationship is

$$y=f(x_1,...,x_k)$$

with $x_1,...,x_k$ factors explaining the $y$. Then first order Taylor approximation of $f$ around zero is:

$$f(x_1,...,x_k)=f(0,...,0)+\sum_{i=1}^{k}\frac{\partial f(0)}{\partial x_k}x_k+\varepsilon,$$

where $\varepsilon$ is the approximation error. Now denote $\alpha_0=f(0,...,0)$ and $\alpha_k=\frac{\partial{f}(0)}{\partial x_k}$ and you have a regression:

$$y=\alpha_0+\alpha_1 x_1+...+\alpha_k x_k + \varepsilon$$

So although you do not know the true relationship, if $\varepsilon$ is small you get approximation, from which you can still deduce useful conclusions.

Answer 2

The other side of the answer, complementary to mpiktas's answer but not mentioned so far, is:

"They don't, but as soon as they assume some model structure, they can check it against the data".

The two basic things that could go wrong are: The form of the function, e.g. it's not even linear in logs. So you'd start by plotting the an appropriate residual against the expected values. Or the choice of conditional distribution, e.g. the observed counts overdispersed relative to Poisson. So you'd test against an Negative Binomial version of the same model, or see if extra covariates account for the extra variation.

You'd also want to check for outliers, influential observations, and a host of other things. A reasonable place to read about checking these kinds of model problems is ch.5 of Cameron and Trivedi 1998. (There is surely a better place for epidemiologically oriented researchers to start - perhaps other folk can suggest it.)

If these diagnostics indicated the model failed to fit the data, you'd change the relevant aspect of the model and start the whole process again.

Answer 3

An excellent first question! I agree with mpiktas's answer, i.e. the short answer is "they don't, but they hope to have an approximation to the right model that gives approximately the right answer".

In the jargon of epidemiology, this model uncertainty is one source of what's known as 'residual confounding'. See Steve Simon's page 'What is residual confounding?' for a good short description, or Heiko Becher's 1992 paper in Statistics in Medicine (subscription req'd) for a longer, more mathematical treatment, or Fewell, Davey Smith & Sterne's more recent paper in the American Journal of Epidemiology (subscription req'd).

This is one reason that epidemiology of small effects is difficult and the findings often controversial - if the measured effect size is small, it's hard to rule out residual confounding or other sources of bias as the explanation.

Answer 4

There is the famous quote "Essentially, all models are wrong, but some are useful" of George Box. When fitting models like this, we try to (or should) think about the data generation process and the physical, real world, relationships between the response and covariates. We try to express these relationships in a model that fits the data. Or to put it another way, is consistent with the data. As such an empirical model is produced.

Whether it is useful or not is determined later - does it give good, reliable predictions, for example, for women not used to fit the model? Are the model coefficients interpretable and of scientific use? Are the effect sizes meaningful?

Answer 5

The answers you have already gotten are excellent ones, but I'm going to give a (hopefully) complementary answer from the perspective of an Epidemiologist. I really have three thoughts on this:

First, they don't. See also: All models are wrong, some models are useful. The goal is not to produce a single, definitive number that is taken as the "truth" of an underlying function. The goal is to produce an estimate of that function, with a quantification of the uncertainty around it, that is a reasonable and useful approximation of the underlying function.

This is especially true for large effect measures. The "take away" message from a study that finds a relative risk of 3.0 isn't really different if the "true" relationship is 2.5 or 3.2. As @onestop mentioned, this does get harder with small effect measure estimates, because the difference between 0.9, 1.0 and 1.1 can be huge from a health and policy standpoint.

Second, there's a process hidden in most Epidemiology papers. That's the actual model selection process. We tend to report the model we ended up with, not all the models we considered (because that would be tiresome, if nothing else). There are a slew of model building steps, conceptual diagrams, diagnostics, fit statistics, sensitivity analysis, swearing at computers and scribbling on white boards involved in the analysis of even small observational studies.

Because while you are making assumptions, many of them are also assumptions you can check.

Third, sometimes we don't. And then we go to conferences and argue with each other about it ;)

If you're interested in the nuts and bolts of Epidemiology as a field, and how we perform out research, the best place to start is probably Modern Epidemiology 3rd Edition by Rothman, Greenland and Lash. It's a moderately technical and very good overview of how Epi research is conducted.