Goodness of fit, predictive power, discrimination

Question

I'm making a couple of logistic regression based predictive models and intend to compare them and see which is "best". "best" here is obviously ill-defined, but as I'm looking for common metrics for evaluating the model performance I've came across different uses of GoF. For instance, on this wiki page R^2s are GoF, while here R^2s are measures of "predictive power". I also read somewhere that ROC is a GoF measure.

So my question is, is there a distinction between GoF and predictive power? And is discrimination something that can be under either? Also, where does proper score functions, c-statistic and specificity/sensitivity fall under?

Answer 1

One way to look at this issue is that goodness of fit is training error and predictive accuracy is test error. ("Predictive power" is not a very precise term.) That is, goodness of fit is how well a model can "predict" data points you've already used to estimate its parameters, whereas predictive accuracy is how well a model can predict new data points, for which it hasn't yet seen the true value of the dependent variable. Many of the same metrics, such as root mean square error, can be used to quantify goodness of fit as well as predictive accuracy; what distinguishes the two cases is whether the model has been trained with the data in question.

Which is more important? Personally, I care a lot more about predictive accuracy. This tells you how useful the model would be for predicting unseen data in the future. Goodness of fit is what you should pay attention to if you think of the model as purely descriptive, as providing a summary of the data, rather than predictive. To be clear, the model with the best fit may not be the most predictively accurate, and vice versa, so there's a real choice to be made here.

Now, often, data analysis is done for explanatory reasons, where the researcher isn't interested in describing the data or predicting new observations so much as making inferences about the true underlying data-generating process, that is, the explanation for the data. Whether goodness of fit or predictive accuracy is better for this is unclear, not least because neither does a particularly good job of saying how accurate the model is as an explanation. My opinion is that goodness of fit is better, but it's clear that mindlessly trying to optimize goodness of fit, without regard for content-specific issues, won't get you to good explanations fast. Explanation is ultimately a less statistical and more scientific concept than goodness of fit or predictive accuracy.

Answer 2

Just to add my two cents, the goodness of fit is how well your data fits the assumptions used to make the model. For example if you are fitting your data to a linear regression model, the goodness of fit answers the question does your data fit a linear regression model according to the assumptions used to create a linear regression model? If the answer is no, then you will certainly have problems with the statistical inference of the regression coefficients amongst other problems.

The goodness of fit can be done by visualizing the data(looking the the QQ plots and various other plot for the normality assumptions ( for linear regression and even for logistic regression), or by performing goodness of fit test Hypothesis Testing Procedure

𝐇_𝟎: the model fits the data vs

𝐇_𝐚: the model does not fit the data

Over here the test statistic is [1]: https://i.stack.imgur.com/L1wm0.png

And the test statistic should follow a chi-squared distribution to pass the 𝐇_𝟎.

My recommendation is that the data or test data should pass the goodness of fit test before going on to the prediction or predictive power test. Otherwise you can get a model that doesn't fit your assumption that the model is based upon. This might be ok if you are NOT interested in the meaning of the coefficients of the model or you not interested in prediction out of the population of your dataset.

The predictive power is how well can your model predict data it has not seen. Various matrices can be used to judge the predictive power 1.Mean squared prediction error (MSPE)

2.Mean absolute prediction errors (MAE)

3.Mean absolute percentage error (MAPE)

4.Precision error (PM)

5.Confidence Interval error (CIM)

6.You can even use R2 or adjusted R2

You can use on of these matrices to select one among many models.

In summary the goodness of fit test makes sure the model you are using align with the assumptions used to generate that model. The predictive power is what it is: how well your model accounts for the variation of of the observations of errors in the data set. So you can get a model that doesn't pass the goodness of fit test, but does predict very well.