How does one compare the statistical performance of different models on a regression or function approximation task over different data sets?

Question

I wanted to compare and potentially justify that one model is better than another on function approximation or (regression task). The issue that I have is that the interpretability of what "good" means in function approximation seems ill defined or at least harder to compare in a absolute sense since the scale for error in function approximation is arbitrary.

It seems to me that this problem does not exist in classification tasks or at least its less severe. For example, if I have two models and I show that their difference in classification accuracy is around ~5% in 10 different classification tasks (or the classification accuracy difference is consistent), then its clear which one is better and by how much (or if the difference is "big"). However, function approximation tasks have the unfortunate property that function approximations lacks this sense of absolute correctness because it does not have a sense of "classification error". For example if one uses the l2 loss and uses 10 different functions to approximate, the error might be 10 while the error in another might be 0.1 but this difference might be an issue of scale.

Is there a way to compare quantitatively how two different models do on function approximation across different data sets?

---

Some ideas:

The natural idea is to find a "normalization" or standardization of the function. However, its unclear to me what type of normalization would be sensible for function approximation since it would be nice to not "screw up" the regression task because of normalization. Also, its not clear to me how to normalize across different data sets so that the sense of small error is the same across all the data sets.

A different idea that seems rather hacky is to just force the task to be a classification task. For example, we could bin the function values across say 10 bins and see what the "classification" of the learning algorithm is. I was thinking to still train the algorithm on the l2 loss and then test the classification error, though its unclear if this is a sensible idea (since what we truly want is the functions to be close, not to have high classification error in this weird arbitrary new task).

Or is the best thing one can do is first restrain the x axis of the function (say to a hyper cube) and then make sure the function is always rescaled such that its between -10 and 10 (scaled in that way so that if we get a score of zero, that is more likely to mean “good” error).

I've also been suggested to divide by the variance/standard deviation of the co-domain (i.e. divide by the standard deviation of $f$). As in:

$$ f(x_i) = y_i \rightarrow \frac{ f(x_i)}{\sqrt{ Var[Y]} } \approx \frac{y_i}{ \sqrt{y_i - \mu_y ]} }$$

where $ \mu_y $ is the empirical mean. I (sort of) see the intuition on doing this but for some reason I'm not 100% if its a sound method. For example, if the data was Normal subtract the mean and divide by the std making it N(0,1). This might makes more sense if that were true however, if the data doesn't follow that structure which it might not. To be precise, most of my data I generate by first sampling a hyper cube $ [-1,1]^D = X$ uniformly and then to each data (domain) point $x_i$ in the data set apply the function to be learned $f(x_i)$). Then I have the data set $S = \{ (x_i, y_i = f(x_i) )\}^N_{n=1}$. I am not sure if standardizing the data might make sense in this context. Does it make sense? Is the data normal? Even if it were normal I'm not sure if it makes sense.

Answer 1

Similarly as people do in classification tasks you could compare your predictions to a random chance baseline.

Now, in classification this is usually done implicitly: people know AUC equal to $0.5$ or accuracy equal to $1/k$ means a result no better than random chance.

In regression tasks one random result would be outputting the mean value, which means the squared error would be equal to the variance of the data. So there you have it, you can normalize by the variance.

So, define a relative measure, which directly compares to the non-informative constant scenario. Here I show you the Root Relative Squared Error, just as shown in Normalized RMSE:

$$RRSE = \sqrt{ \frac {\sum{(x_{p,i} - x_{a,i})^2}} {\sum{(x_{a,i} - \overline x_{a})^2}} }$$

$x_{a,i}$ is the $i$-th actual value and $x_{p,i}$ is the $i$-th predicted value.

If $RRSE \geq 1$ your predictions aren't better than simply outputting the mean value, which brings us an intuition analogous to what we get in classification evaluation metrics.

There are other relative measures as well, and keep in mind they will fail if in reality the spread of the true values is zero, i.e. they are constant.