I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so TSS<>RSS+ESS. Trying to compute R^2 results in a negative number sometimes - R^2 is intended for linear models. Any thoughts of the best way to get a statistic for this? I can do mean absolute deviation to compare two different models, or take the deviation and square it (essentially RSS), but that mostly just tells me if a model is better or worse than another one, not necessarily "good" or "bad." With an R^2 of .75 for my purposes I can say that the fit of one model is pretty good. Is there a better statistic than R^2 (or adjusted R^2 - they suffer from the same problem) I can get where I just have actual values and the predicted values and make an evaluation based on that?
There are a few options, but none of them give an "absolute truth" result between 0 and 1. The two common options are AIC and BIC. see here: Model selection with nonlinear fitting? Statistical tests seem ambiguous
Another option is the S measure: http://blog.minitab.com/blog/adventures-in-statistics/regression-analysis-how-to-interpret-s-the-standard-error-of-the-regression
- As models become more complex, there are less "absolute truth" answers like R^2 between 0 and 1. With more flexible models, it's easy to get to a 100% fit of training data, and metrics like R^2 are irrelevant.
- Make sure you test your model with some test data. non linear models tend to overfit.