When performing GAM and GLM fits to the same data set, I get an almost identical fit in terms of fitting metrics. However, the variables that are identified as significant differ between the two approaches. How do decide which method is most appropriate?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
13
The reason that the significance of the coefficients differs is because, under the modeling assumptions, the calculation of standard errors differs. The correct answer is to choose the model that is most appropriate for--not the data--the scientific question at hand.
For instance, suppose I am interested in measuring a difference in rates of some disease in communities before and after applying some intervention. I could model the additive rates using a Poisson GLM with the identity link. This presumes that the distribution of rates follows a Poisson distribution. On the other hand, I could model these rates using a simple t-test. This makes no assumption about the distribution of rates and accounts for the possible underdispersion due to infections within a community not necessarily having a constant inter-arrival time. These two models will estimate the same rate difference, but give different standard errors.
Which, then, is correct? Well, the adequacy of the underpinning assumptions is what is called into question. Therefore, you should endeavor to describe and rationalize the model that is appropriate for the data--whether or not that model happens to be significant.
-
1$\begingroup$ I'm not sure it's quite correct to say that a t-test "makes no assumption about the distribution"; how then does one derive the t-distribution that is used to calculate the p-values? $\endgroup$– Glen_bMar 5, 2015 at 0:51
-
$\begingroup$ Thanks for the reply, AdamO. What I am aiming to achieve is to identify the most significant variables, but I don't know what the distribution is or should be - Poisson, Gaussian, or other. When fitting a GAM or GLM there are points of agreement, however (e.g., one variable always is most significant). In the absence of knowledge of the distributions, I defaulted to Gaussian & identity and assumed that if GAM and GLM give the same results, it means I should go with GLM. I have no reason to think it's not Gaussian, but that isn't great rationale. Do I try many different distributions? $\endgroup$– DanMar 5, 2015 at 3:08
-
$\begingroup$ @Glen_b In two steps: by way of the Central Limit Theorem and also by way of using the T-approximation to the normal distribution when the standard deviation is estimated using the sample mean. The independence of events is an important assumption as well as having a sufficient sample size so that the T approximation to the sampling distribution of the test statistic is "good enough". This is usually a modest number. $\endgroup$– AdamOMar 5, 2015 at 19:07
-
1$\begingroup$ @Lyngbakr you say that you aim to "...identify the most significant variables". This is not an aim, it is a statistical fishing expedition. In both exploratory and confirmatory analyses, it is very useful for the statistician to work directly with the investigators to prespecify a set of hypotheses. $\endgroup$– AdamOMar 5, 2015 at 19:09
-
$\begingroup$ AdamO The t-statistic consists of a numerator and a denominator, so an argument about the numerator alone isn't enough to establish normality of the ratio (though Slutsky's theorem can be applied), and then the result is only in the limit as $n\to\infty$; to say anything about finite sample sizes you'd need to use something like Berry-Esseen. ...(ctd) $\endgroup$– Glen_bMar 5, 2015 at 21:11