Leave-one-out cross validation in regression: R squared cannot be used - how else may model performance on unseen data be evaluated?

Question

I have a regression problem with very few datapoints, therefore I want to use leave-one-out cross validation (effectively N-fold cross validation with N being the number of datapoints) to determine model performance on unseen data. I would like to compute the coefficient of determination $R^2$ because of its straightforward interpretation as the variance explained which clearly indicates how well the model will perform on new data. However, since I have only one predicted target value for each of the N folds, $R^2$ is zero unless the predicted value coincides exactly with the true target value. This seems to render the application of $R^2$ useless in this case. Are there other metrics I may use with a similar straightforward interpretation? What other ways exist to approach this problem?

P.S.: One alternative I came up with is computing $R^2$ using the set of all predicted target values (from all N folds) and the true target values. However, this leaves me without an estimation of its variance.

Answer 1

(This seems to be a near-duplicate of a question I answered a year ago.)

$R^2$ is often defined as a comparison of the sum of squared residuals for the model of interest vs the sum of squared residuals for a model that only has an intercept. With this in mind, I would proceed as follows.

1. Fit models using the leave-one-out cross-validation strategy.

2. However, for each data set with a point left out, fit two models: your model of interest and an intercept-only model.

3. For each obseration that is left out, your two models will give predicted values for the observation that is left out. Take the difference between that prediction and the true value of the observation that was left out, and square that value.

4. Across all of the obserations, you get one of these squared residuals. Add up the squared residuals from your model of interest; denote this as $L_1$. Now add up the squared residuals from your intercept-only models; denote this as $L_0$.

5. Calculate an $R^2$-style statistic as $1 - \dfrac{L_1}{L_0}$, which is extremely analogous to the usual $R^2$.

Proceeding this way allows you to do an $R^2$-style calculation with only one observation in each test set while also having an interpretation as the PRESS statistic for your model of interest compared to the PRESS statistic for an intercept-only model. If your model of interest cannot outperform (in terms of PRESS) an intercept-only model, which will be denoted by $1 - \dfrac{L_1}{L_0}< 0$, then your model has some issues.