I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so TSS<>RSS+ESS. Trying to compute R^2 results in a negative number sometimes - R^2 is intended for linear models. Any thoughts of the best way to get a statistic for this? I can do mean absolute deviation to compare two different models, or take the deviation and square it (essentially RSS), but that mostly just tells me if a model is better or worse than another one, not necessarily "good" or "bad." With an R^2 of .75 for my purposes I can say that the fit of one model is pretty good. Is there a better statistic than R^2 (or adjusted R^2 - they suffer from the same problem) I can get where I just have actual values and the predicted values and make an evaluation based on that?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
There are a few options, but none of them give an "absolute truth" result between 0 and 1. The two common options are AIC and BIC. see here: Model selection with nonlinear fitting? Statistical tests seem ambiguous
Another option is the S measure: http://blog.minitab.com/blog/adventures-in-statistics/regression-analysis-how-to-interpret-s-the-standard-error-of-the-regression
Some thoughts:
- As models become more complex, there are less "absolute truth" answers like R^2 between 0 and 1. With more flexible models, it's easy to get to a 100% fit of training data, and metrics like R^2 are irrelevant.
- Make sure you test your model with some test data. non linear models tend to overfit.
-
$\begingroup$ So, I don't really have a model to work with - I just have two vectors of values that are provided to me, so I can't really compute the likelihood function or anything, so AIC and BIC are probably a bit tricky. I was having trouble finding a great solution. It sounds like there are a few different options, but that there is not necessarily a standard, great way to approach it. $\endgroup$ Commented Feb 11, 2015 at 20:46
-
$\begingroup$ Not that I'm familiar with, but maybe other people can help. By the way, if you got a negative R^2, you probably have a bug somewhere... $\endgroup$– omri374Commented Feb 11, 2015 at 20:53
-
$\begingroup$ Instead of trying to evaluate how good is your model, try to evaluate different models on a test set. This way you'll be able to see which model predicts an unknown sample in a better way. Especially if your task is prediction. $\endgroup$– omri374Commented Feb 11, 2015 at 20:54