1
$\begingroup$

The response values in my data set (100 data points) are all positive integers (should not be either negative or zero values). I have developed two statistical models: Linear Regression (LR) and K Nearest Neighbor (KNN, 2 neighbours) using the data set in R. The R methods I have used are lm() and knn.reg(). To select between these two models, I have conducted 10 fold cross-validation test and first computed root mean squared error (RMSE). Although the LR model is giving negative prediction values for several test data points, its RMSE is low compared to KNN. When I see the prediction values of KNN, they are positive and for me it makes sense to use KNN over LR although its RMSE is higher. Moreover, when I used Normalized RMSE (http://en.wikipedia.org/wiki/Root-mean-square_deviation), KNN has low NRMSE compared to LR.

Furthermore, I would like to define "prediction accuracy" of the models as (100 - NRMSE) as it looks like we can consider NRMSE as percentage error. Please let me know the above methodology I am following is fine or not. Thank you.

$\endgroup$
4
$\begingroup$

There seem to be at least two distinct questions intertwined here.

First is the question of the right model for your data. As your response is, and can only be, positive integers it seems unlikely that linear regression by itself is a suitable choice because, as you have found, it may predict impossible values: the choice of figure of merit or error metric is by comparison quite secondary. You have various alternatives open to you, including working with a logarithmic transformation. My top suggestion would be to check out Poisson regression. In R that can be done using glm() and quite possibly in other ways. (R experts may well add much more.) See for an introduction

http://en.wikipedia.org/wiki/Poisson_regression

and for one engaging discussion see

http://blog.stata.com/tag/poisson-regression/

The Stata content of that blog does not render the posting useless or uninteresting to people who don't use Stata.

Poisson regression can only predict positive values. (Those predictions can be fractional, to be understood in exactly the same spirit as statements that the mean number of children per household is 1.2, or whatever.)

The second question is about RMSE and NRMSE. The merit of RMSE is to my mind largely that it is in the same units of measurement as the response variable. Statisticians and non-statisticians should find it relatively easy to think in terms of RMSE of 3.4 metres or 5.6 grammes or 7.8 as a count.

Naturally, nothing stops you scaling it and it then loses that interpretation and becomes a relative measure. It is just what it is and joins a multitude of other such measures, e.g. R-square and its many pseudo-relatives, (log-)likelihood and its many relatives, AIC, BIC and other information criteria, etc., etc. The choice of figure of merit, error metric or of whatever you call them -- if I recall correctly Bowley wrote of "misfit" in 1902; that's a nice word worthy of use -- is partly a matter of personal taste, partly a matter of audience (only technical audiences can be expected to recognise AIC, for example), and mostly a matter of what has become conventional in your field.

$\endgroup$
  • $\begingroup$ Thank you. I have used AIC for selecting important predictors of my models using stepAIC() method in R. We can compute AIC of the linear regression model, but I got errors when I applied R AIC() method on the KNN object. Could you tell me how to get AIC() value on the KNN object. $\endgroup$ – samarasa May 24 '13 at 14:02
  • $\begingroup$ How do you get log likelihood out of KNN? I think you need to start a separate question, as you are asking something quite different. $\endgroup$ – Nick Cox May 24 '13 at 14:28
  • $\begingroup$ Done. Please see at stats.stackexchange.com/questions/59946/… $\endgroup$ – samarasa May 24 '13 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.