Are goodness of fit and lack of fit the same?
If they are tests, is one's null the other's alternative?
$\begingroup$
$\endgroup$
4
-
3$\begingroup$ No, they are exact opposites. The null is the same, which is obvious: it's impossible to establish a null distribution when the null and alternative are switched. $\endgroup$– whuber ♦Aug 2, 2013 at 2:22
-
$\begingroup$ @whuber: Thanks! How are they opposites? What are their nulls and alternatives? $\endgroup$– TimAug 2, 2013 at 2:42
-
3$\begingroup$ I'm sorry, you're not going to pull me into one of these endless debates. The answers to your question are obvious. $\endgroup$– whuber ♦Aug 2, 2013 at 2:43
-
2$\begingroup$ @whuber: I am not pulling you to endless debates. I am really wondering. $\endgroup$– TimAug 2, 2013 at 2:46
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This is more about use of English within statistical science than anything deeply or problematically technical.
Talking about "goodness of fit" goes back at least to Karl Pearson in 1895 (see http://www.stat.iastate.edu/preprint/articles/2011-10.pdf for that and other historical attributions).
The main idea now is simply, in terms that Pearson would not have used, How well does a model fit?
However, it makes equal sense to ask, How badly does a model fit? The term "badness of fit" has been used occasionally (e.g. by Joseph B. Kruskal), but it is swamped in literature I know by "lack of fit".
There is no clash of concepts here, no more than between How full is your glass? and How empty is your glass? It is just a case of which way round we view the question, positively or negatively.
Goodness of fit or lack of fit is something we can assess graphically. For example, experienced researchers are often quite happy after looking at a normal probability plot that a distribution is close enough to normal to be unproblematic for their purposes.
One common reason for using different terminologies, I think, lies in a nuance arising with numeric measures of goodness of fit.
Take for example a chi-square statistic for a standard problem such as independence of rows and columns in a two-way contingency table. Here (for a given number of degrees of freedom) the higher the chi-square statistic, the worse the fit. So, as a matter of English, correctness according to some and pedantry according to others, it could be objected that chi-square doesn't measure goodness of fit, it measures badness of fit or lack of fit. Some would be happy to meet the nuance by emphasising that chi-square is an inverse measure of goodness of fit, just as (say) inflation and unemployment may be considered inverse measures of the health of an economy.
Another example comes from linear regression. Here two among several possible figures of merit are the coefficient of determination $R^2$ and the root mean square error (RMSE), or (loosely) the standard deviation of residuals. $R^2$ could be described as a measure of goodness of fit in both weak and strict senses: it measures how good the fit is of the regression predictions to data for the response and the better the fit, the higher it is. The RMSE is a measure of lack of fit in so far as it increases with the badness of fit, but many would be happy with calling it a measure of goodness of fit in the weak or broad sense.
(The expression "figure of merit" slipped in here as alternative wording. It seems most common in some areas of physical science as a general term for some single-number indicator of how well a method did.)
I don't think we need spell out, or discuss, anything to do with null or alternative hypotheses here. Indeed, the terms are applicable in situations where there is no hypothesis testing at all.
I leave untouched the immensely more important question of how well any measure of discrepancy between model and data serves as a measure of goodness, badness or lack of fit, and how to choose between competing measures. Often different measures measure different things are complementary in value: $R^2$ and RMSE in regression are arguably a good example. At the same time, calculating numerous different measures and cherry-picking the most congenial or the most favourable should be avoided.
$\begingroup$
$\endgroup$
The problem is that within the frequentist framework, for goodness-of-fit tests there is no such symmetry.
This is because P-values have to be interpreted as evidence against a model/hypothesis, so if you find a P-value larger than 0.5, you are technically in a limbo.
Personally I prefer to interpret them as lack thereof.
The RSquared interpretation is rather different, but using out-of-sample measures and information criteria based on the Likelihood, you could have more support for a specific hypothesis than another (model with 1 regressor VS model with 2 regressors)
