How to determine when the difference between two goodness of fit values is significant?

I have a data series that I'm trying to fit to a model. I'm trying several types of models (exponential, linear, logarithmic). In order to assess which one it fits best, I use a Residual Sum of Squares as a goodness of fit measure. In some cases the difference between the RSS is very small. I was wondering if there is any statistically significant way of saying that the difference one RSS is significantly smaller that the other RSS in order to justify saying that the data definitely fits a certain growth model. Forgive my lack of statistical terminology if any. My background is in CS.

• It sounds like you may be looking for something like a Likelihood Ratio Test? That wouldn't use the RSS, but it would tell you which model fits better, given the difference in the number of variables. Another option would be to use leave-k-out cross validation and calculate the RSS if you predict each value from a sample fit to the other samples. Jan 28, 2016 at 15:42
• RSS is not going to be suitable if you have different numbers of parameters in some models. $\qquad$ $\:$ @Jautis likelihood ratio tests would require nested models. Jan 28, 2016 at 16:39
• @Glen_b That's why I thought cross validation would work if they're not nested models. Jan 28, 2016 at 17:48
• @Jautis: Thanks I will look into the option of the leave-k-out cross validation. Jan 28, 2016 at 23:45
• @Glen_b: That is good to know. Could you please point me to some resource that has some details on why this happens? Jan 28, 2016 at 23:46

1 Answer

If your problem is "simple enough" a very effective way to do this is using the "Bayesian Information Criteria". It is defined as:

$$BIC=\chi^2+df\times ln(n)$$.

Where $$df$$ is the degrees of freedom of the model (number of points - number of fitted parameters) and $$n$$ is the number of data points you have. $$\chi^2$$ is basically the residual sum of square divided by $$n$$ (the number of points), if the variance of each data point is identical. If not, divide each squared residual by the variance of each point.

If a model gives you a BIC which is at least 2 lower than another model, then the new model is better.

• Both BIC and AIC (Akaike Information Criterion) are used in this way to evaluate which of a set of models is superior. Technically, however, I don't believe that there is a way to use these measures that can demonstrate a "statistically significant" difference among models.
– EdM
Jan 28, 2016 at 18:05
• @user26067: Thank you. This looks like what I could use, especially as it does not look too computationally intensive. This answer is probably most suitable for my needs. Do you want me to go ahead and accept it, or wait for some more answers and choose the best (regardless of my needs)? Jan 28, 2016 at 23:49
• @EdM, I'm not 100% sure about the theory, but you can make a rule of thumb calculation proving that, under normality assumptions, a decrease of less than 2 in BIC can be attributed to random fluctuations alone. So, a decrease bigger than that it's definitely worth noticing. Jan 29, 2016 at 9:22