1
$\begingroup$

I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so TSS<>RSS+ESS. Trying to compute R^2 results in a negative number sometimes - R^2 is intended for linear models. Any thoughts of the best way to get a statistic for this? I can do mean absolute deviation to compare two different models, or take the deviation and square it (essentially RSS), but that mostly just tells me if a model is better or worse than another one, not necessarily "good" or "bad." With an R^2 of .75 for my purposes I can say that the fit of one model is pretty good. Is there a better statistic than R^2 (or adjusted R^2 - they suffer from the same problem) I can get where I just have actual values and the predicted values and make an evaluation based on that?

$\endgroup$
1
$\begingroup$

There are a few options, but none of them give an "absolute truth" result between 0 and 1. The two common options are AIC and BIC. see here: Model selection with nonlinear fitting? Statistical tests seem ambiguous

Another option is the S measure: http://blog.minitab.com/blog/adventures-in-statistics/regression-analysis-how-to-interpret-s-the-standard-error-of-the-regression

Some thoughts:

  1. As models become more complex, there are less "absolute truth" answers like R^2 between 0 and 1. With more flexible models, it's easy to get to a 100% fit of training data, and metrics like R^2 are irrelevant.
  2. Make sure you test your model with some test data. non linear models tend to overfit.
$\endgroup$
  • $\begingroup$ So, I don't really have a model to work with - I just have two vectors of values that are provided to me, so I can't really compute the likelihood function or anything, so AIC and BIC are probably a bit tricky. I was having trouble finding a great solution. It sounds like there are a few different options, but that there is not necessarily a standard, great way to approach it. $\endgroup$ – theHamburglar Feb 11 '15 at 20:46
  • $\begingroup$ Not that I'm familiar with, but maybe other people can help. By the way, if you got a negative R^2, you probably have a bug somewhere... $\endgroup$ – omri374 Feb 11 '15 at 20:53
  • $\begingroup$ Instead of trying to evaluate how good is your model, try to evaluate different models on a test set. This way you'll be able to see which model predicts an unknown sample in a better way. Especially if your task is prediction. $\endgroup$ – omri374 Feb 11 '15 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.