I'm investigating associations between socioeconomic factors and dichotomous outcome. I use generalised linear models (GLM) with log link for Bernoulli family, i.e., modelling the prevalence ratio. At the epidemiology course of K.J. Rothman & E.Hatch we were told, that goodness of fit tests are designed to prediction models and in causal inference it is not important with model fit. I can not find any reference on that. Can anybody comment on this and suggest a reference? Thank you in advance!
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
10
This is correct. In the epidemiology or social science, we would like to find the causal association between, say exposure and outcome. Then the most important thing is to identify the confounding factors, which need to be adjusted in your multivarate model settings. This does not necessarily mean to fit a model well, but only for adjustment purpose which makes ur estimate of interest unbaised due to other factors associated with outcome as well. For example, if we want to study the assocaiton between lung cancer and heavy drinking, then smoking status has to be adjusted as a confounding variable. Because smoking has been recognized as a risk factor of lung cancer. Therefore the heavy drinking status is confounded to the smoking, probably because heavy drinkers usually a smoker too. This is usually the utmost important consideration behind medical research.
If your purpose is for prediction, then you dont need to think confounding at first place, and you can include interaction terms, 2 ways, three ways interactions as a model budiling procedure, testing goodness-of-fit, and etc. And algorithms such as forward/backward selection are valid to provide a good model.
If you purpose is to provide a valid measure of an effect, then those selections are not quite appropriate. Because even a variable is not significant in a model, it still might be kept in the model such as age and gender, which are always adjusted in those epidemilogy study. Also the interaction effect might or might not be of interest in the epidemiology study.
Chapter 6 in the book of "Logistic Regression A Self-learning Text" provides a detailed explanation of model building strategy for what you asked about.
Cheers.
-
5$\begingroup$ I don't understand this reasoning. It seems that if you have a bad fit, then you could get the causal association completely wrong. Hormesis is a good example that should be important to epidemiology: if you fit a linear dose-response curve to a hormetic effect, you not only wind up overlooking the hormesis, but you draw entirely the wrong conclusions about low-level doses. How, then, can one justify overlooking this possibility by not checking goodness of fit? Am I misunderstanding your claims? $\endgroup$– whuber ♦Commented Dec 1, 2011 at 16:45
-
1$\begingroup$ I must say i totally disagree with this approach, you always want a good model fit, for it indicates that your "causal" variable has been entered in the right way, and that it is not merely picking up random correlarions between causal variables which haven't been observed. also a poor goodness of fit makes it very difficult to say "A causes B" when there is so much extra things going on with the part of the data which didn't fit. $\endgroup$ Commented Jan 31, 2012 at 10:34
-
1$\begingroup$ @whuber It's an identification issue. The following minimal example might help: A and B independently cause C, i.e. A -> C <- B. A does not cause B so the corresponding regression coef of B on A should, if we want to interpret as a causal effect, be 0. But models also conditioning on C will all fit better than those that do not and also give a non-zero coef. Nevertheless this will be 'causally' wrong because this coef is the association but isn't the causal effect. And yes, from the data we cannot confirm the "A -> C <- B" part that grounds this argument. $\endgroup$ Commented Mar 1, 2012 at 9:58
-
2$\begingroup$ @Conjugate Thank you for the clarification! I'm still not understanding, though, how this consideration would justify accepting lack of fit in a model. It seems to me the correct message is that we should use an appropriate model in the first place and interpret lack of fit as evidence that our model provides inadequate explanatory power (and therefore could give completely wrong results). After all, if you don't care about lack of fit, then nothing can falsify your model. You're no longer doing science--you're into fairy-tale and ESP land. Or so it seems to me... $\endgroup$– whuber ♦Commented Mar 1, 2012 at 15:28
-
2$\begingroup$ @whuber Following up on my tiny example: The idea is not quite that nothing can falsify the model but that nothing in the data set alone can distinguish between the alternatives A <- C -> B and A -> C <- B, despite the fact that you should and should not condition on C respectively. However, manipulating C and watching A and B will tell you which structure you have and therefore whether you should condition on C. But under all circumstances a model with C in it should look better than one without. $\endgroup$ Commented Mar 2, 2012 at 16:03
$\begingroup$
$\endgroup$
It is simpler to think in terms of linear models. The same logic holds with logit and with nonlinear models, though it is more complicated. For causal interpretation, you need a bunch of assumptions to hold. I might be missing some.
Here is your model: $$ y = \delta T + X\beta + \epsilon $$ $T$ is the variable you care about, $X$ are covariates. $\epsilon$ is everything else that causes $y$.
Let's say that $y$ is Donald Trump's logged daily Diet Coke consumption, in milliliters. $T$ is a dummy variable for Mueller being in the news, and $X$ are numerous controls such as school shootings, the stock market, whether the president's daily brief had too many big words in it, and whether #MAGA is trending on twitter.
Now, Donald Trump will drink lots of diet coke when he is stressed out, and other things make him stressed. For example, Rachael Maddow. Now, Rachel Maddow's general relevance to Donald Trump is highly correlated with Mueller being in the news. But let's say that the staistician doesn't get MSNBC. Rachael Maddow is unobserved -- therefore it is one component of $\epsilon$. So the main assumption is
Number 1 $$ E[\epsilon|T,X] = 0 $$ This means that there is nothing that you don't observe that has any effect on $y$, that is also correlated with $T$ or $X$. (Actually, it only needs to be $E[\epsilon|T] = 0$ when $T$ is orthogonal to $X$.)
Back to our example: Rachael Maddow is unobserved to the statistician, is a part of $\epsilon$, and is correlated with $T$. That means that $\epsilon$ won't be zero in expectation, and your estimate is confounded.
What can you do? Either get MSNBC, or find some sort of identification strategy to deal with your confounding problem. Identification strategies are econometric methods of establishing causality when a naive regression will be biased because of confounding, usually in observational data.
There are some others, having to do with mis-specification, etc. But they are rather in the weeds and I'm out of time.
Regarding goodness of fit: you do not need goodness of fit to identify the expectation of Donald Trump's coke drinking given Robert Mueller is in the news, given a sufficiently big sample size. You just need your assumptions to hold. Usually, they will not hold, which is why people think that you need a well-fitting model. After all, a model where $\epsilon = 0$ in the population is one that can be interpreted causally.
answered Mar 1, 2018 at 22:59
$\begingroup$
$\endgroup$
From Kleinbaum and Klein 2010, logistic regression, p304.
In most epidemiologic analyses, the primary goal is to assess an exposure–disease relationship, so that we are usually more interested in deriving the “best” model for the relationship (which typically involves a strategy requiring the comparison of various models) than in using a GOF procedure. Nevertheless, once we have obtained a final (i.e., best”) model, we would also like this model to fit the data well, thus justifying a GOF procedure
This is my reasoning: We study asociation through ORs (Sometimes RR) which are related to probabilities. With probabilities you predict. If your predictions are too bad (there is statistical evidence of lack of fit) so are your probabilities and there for you measure of association (OR) is also a bad one. For me the difference with the prediction goal is the process of fitting the model. In the latter case, we seek maximum prediction power no matter what. In the association case (not causal, as you asked) we seek for valid $\beta$s sacrificing GOT but not too much (to the point where the $\beta$s are useless).