What's a Good $R^2$ Score in K Nearest Neighbors?

Question

I'm fitting SciKit-Learn's KNeighborsRegressor on a 5 dimensional space and my model performance is peaking at a score of $\sim 0$.

In their documentation they say that the score they're using is the following:

$$R^2 = (1 - \frac{u}{v})$$

which I believe is the formula for the Coefficient of Determination, making:

$$u = \sum^N_i(y_{p,i} - y_{t,i}), \\ v=\sum^N_i(y_{t,i} - \bar y_{t,i})$$

where $N$ is the number of samples, $y_{p, i}$ is the predicted value of sample $i$, $y_{t, i}$ is the true value of sample $i$, and $\bar y_{t, i}$ is the mean of the true samples.

This makes $R^2$ the "unexplained variance of the dataset". I'm struggling to understand what that means for my model's performance, and the SciKit-Learn documentation isn't much help:

The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a $R^2$ score of 0.0.

When measuring model performance using the Coefficient of Determination, $R^2$, what is a good score?

Answer 1

In general, it is hard to say what constitutes a good score. I have seen papers in top journals that have $R^2<0.1$. At the same time, it might be the case for a different problem that $R^2=0.9$ is nothing worth celebrating.

An advantage that $R^2$ has over other measures of performance is that it inherently gives some kind of comparison to a baseline model; if you can’t beat the baseline model, your model isn’t helping. This corresponds to $R^2\le 0$, with equality denoting the exact same performance as the baseline model.

Thus, as long as you get $R^2>0$, you’re doing something useful in terms of predictive ability. At the very least, you are performing better than a reasonable baseline model. How much better than $0$ you need to be is going to depend on the problem and how others have performed. If you get the best value anyone has ever gotten, that sounds like good news! If you beat the baseline model but fall short of what most others can achieve, there is room to improve.

As with any machine learning task, watch out for overfitting.