My question is on the details and subtleties of comparing models in non-linear regression.
Situation: I want to find out which function fits a set of data points better. Therefore, I'm looking for some measure to quantify how well each model describes my data. The models I want to compare a non-linear (including exponential terms) and I only have 5 data points. The data points have some uncertainty in y-direction. Taking those into account would be nice but I'm happy without as well. I do the fitting via a least square optimization. The models have different numbers of free parameters (2 and 4). All models seem to give good fits just by looking at them.
Question: Which of the measures (R-squared, chi-squared, SSE, RMSE, AIC, BIC) can I trust most in my situation? What are the differences? Can one combine them in a meaningful way? Is a statement even possible with such little data?
What I know so far: There are the simple sum of squares due to errors (SSE) and the root mean square error (RMSE) which is the root of the SSE scaled by the number of parameters in the model.
The R-squared coefficient tells me how much of the variance in my data can be explained by my model. A high value though gives no guarantee that the model fits the data well. That can also happen when the fundamental form is mis-specified. It also has the problem that it can always be increased by adding more parameters. This is lifted in the case of the adjusted R-squared.
The chi-square makes a statement about the sum of the differences between the expected and observed points. The reduced chi-square takes into account the number of parameters of the model. (But does that mean the reduced chi-squared of models with a different number of parameters are comparable?)
I observe that the SSE and chi-squared give me quite opposite results.
The bayesian and Akaike's information criterion assess the log-likelihood and have a penalty term to avoid overfitting. The penalty term in BIC is larger than in AIC for sample sizes >7. There is also a second order AIC for small sample sizes (n/k<40). For the gaussian special case there is a formula for BIC in terms of RSS (wikipedia).
There is also the minimum description length which is based on the number of bits needed to represent model and data.
BIC and AIC seem to be made for the purpose of comparing models, but I only find examples where they are used for linear regression with large data sets. Also, I'm not really sure how to implement them.
