QQ-Plots is a graphical device to test the goodness of fit between two distributions.

However, it doesn't tell us whether the probability model that I chose overfits on my data.

  • $\begingroup$ the qqplot presents the data. It is not a test. If I had split my data into a test and validation set, then I might look at the qqplot of the errors of each set and compare those. It might tell me useful things about difference in means, spreads, and sample sizes. $\endgroup$ – EngrStudent Mar 27 '16 at 18:17
  • $\begingroup$ When you compare a theoretical distribution and an empirical distribution, we are assessing whether they both line up. It should be linear if there is a high goodness of fit. $\endgroup$ – user46925 Mar 27 '16 at 18:18

A QQ plot does not necessarily tell you goodness of fit to your model, only the theoretical distribution you're fitting (which may or may not be the ultimate model). And, in cases that it is showing you goodness of fit for your intended model then it's unclear how seeing that tells you overfitting. They're not as related as you seem to think. For example, there might be near 0 noise in a set of data and a linear regression fit almost perfectly. That's not necessarily overfitting. Therefore, a high degree of fit wouldn't tell you overfitting by itself.

  • $\begingroup$ I think that GOF can indeed tell us whether a model overfits. I fit a 1000 component GMM on 1000 data points, then it will have perfect GOF but overfit. $\endgroup$ – user46925 Mar 27 '16 at 18:13
  • $\begingroup$ Blogs like r-bloggers.com/…, say that QQ plots can tell us GOF $\endgroup$ – user46925 Mar 27 '16 at 18:15
  • $\begingroup$ And what if a 2 parameter model fits those 1000 data points? GOF by itself isn't sufficient. Further, what if the 1000 component model fits not only your 1000 but 10000 more out of sample points? Is it over fit now? $\endgroup$ – John Mar 27 '16 at 18:16
  • $\begingroup$ Ah, I see better how you're intending QQ-plots unknown... revised. $\endgroup$ – John Mar 27 '16 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy