3
$\begingroup$

I wanted to compare and potentially justify that one model is better than another on function approximation or (regression task). The issue that I have is that the interpretability of what "good" means in function approximation seems ill defined or at least harder to compare in a absolute sense since the scale for error in function approximation is arbitrary.

It seems to me that this problem does not exist in classification tasks or at least its less severe. For example, if I have two models and I show that their difference in classification accuracy is around ~5% in 10 different classification tasks (or the classification accuracy difference is consistent), then its clear which one is better and by how much (or if the difference is "big"). However, function approximation tasks have the unfortunate property that function approximations lacks this sense of absolute correctness because it does not have a sense of "classification error". For example if one uses the l2 loss and uses 10 different functions to approximate, the error might be 10 while the error in another might be 0.1 but this difference might be an issue of scale.

Is there a way to compare quantitatively how two different models do on function approximation across different data sets?


Some ideas:

The natural idea is to find a "normalization" or standardization of the function. However, its unclear to me what type of normalization would be sensible for function approximation since it would be nice to not "screw up" the regression task because of normalization. Also, its not clear to me how to normalize across different data sets so that the sense of small error is the same across all the data sets.

A different idea that seems rather hacky is to just force the task to be a classification task. For example, we could bin the function values across say 10 bins and see what the "classification" of the learning algorithm is. I was thinking to still train the algorithm on the l2 loss and then test the classification error, though its unclear if this is a sensible idea (since what we truly want is the functions to be close, not to have high classification error in this weird arbitrary new task).

Or is the best thing one can do is first restrain the x axis of the function (say to a hyper cube) and then make sure the function is always rescaled such that its between -10 and 10 (scaled in that way so that if we get a score of zero, that is more likely to mean “good” error).

I've also been suggested to divide by the variance/standard deviation of the co-domain (i.e. divide by the standard deviation of $f$). As in:

$$ f(x_i) = y_i \rightarrow \frac{ f(x_i)}{\sqrt{ Var[Y]} } \approx \frac{y_i}{ \sqrt{y_i - \mu_y ]} }$$

where $ \mu_y $ is the empirical mean. I (sort of) see the intuition on doing this but for some reason I'm not 100% if its a sound method. For example, if the data was Normal subtract the mean and divide by the std making it N(0,1). This might makes more sense if that were true however, if the data doesn't follow that structure which it might not. To be precise, most of my data I generate by first sampling a hyper cube $ [-1,1]^D = X$ uniformly and then to each data (domain) point $x_i$ in the data set apply the function to be learned $f(x_i)$). Then I have the data set $S = \{ (x_i, y_i = f(x_i) )\}^N_{n=1}$. I am not sure if standardizing the data might make sense in this context. Does it make sense? Is the data normal? Even if it were normal I'm not sure if it makes sense.

$\endgroup$
  • $\begingroup$ Is there a reason you'd prefer not to use a standard loss function such as mean squared error? $\endgroup$ – Cliff AB Sep 4 '16 at 4:42
  • $\begingroup$ @CliffAB sorry maybe my question wasn't clear (I'll try to clarify). The mean squared error function is fine. However, I've empirically observed that the two models I'm comparing have an error of 10 and 0.1 in two different function approximation tasks. However, its hard to get a sense if there errors mean its a good approximation, special if the function is high dimensional (can't just plot it). Thus, I'm just brain storming ideas of how to solve the problem, since I'm not sure 100% exactly what is causing the problem. $\endgroup$ – Charlie Parker Sep 4 '16 at 4:45
2
$\begingroup$

Similarly as people do in classification tasks you could compare your predictions to a random chance baseline.

Now, in classification this is usually done implicitly: people know AUC equal to $0.5$ or accuracy equal to $1/k$ means a result no better than random chance.

In regression tasks one random result would be outputting the mean value, which means the squared error would be equal to the variance of the data. So there you have it, you can normalize by the variance.

So, define a relative measure, which directly compares to the non-informative constant scenario. Here I show you the Root Relative Squared Error, just as shown in Normalized RMSE:

$$RRSE = \sqrt{ \frac {\sum{(x_{p,i} - x_{a,i})^2}} {\sum{(x_{a,i} - \overline x_{a})^2}} }$$

$x_{a,i}$ is the $i$-th actual value and $x_{p,i}$ is the $i$-th predicted value.

If $RRSE \geq 1$ your predictions aren't better than simply outputting the mean value, which brings us an intuition analogous to what we get in classification evaluation metrics.

There are other relative measures as well, and keep in mind they will fail if in reality the spread of the true values is zero, i.e. they are constant.

$\endgroup$
  • $\begingroup$ Sorry if I don't understand/appreciate, but why is outputting the mean analogous to predicting change? Intuitively, I always thought that maybe outputting the mean is a good idea since that would be on average, the correct prediction to output (since its the mean). $\endgroup$ – Charlie Parker Sep 5 '16 at 6:04
  • $\begingroup$ @CharlieParker It's the last informative prediction, isn't it? For that you wouldn't even need machine learning. So it's a non-informative output, a baseline every model should surpass to be acceptable. $\endgroup$ – Firebug Sep 5 '16 at 16:59
  • $\begingroup$ My intuition stemmed from statistical decision theory. If we want to predict optimally with the square loss (assuming we have the true distribution), I thought the optimal thing to do is to predict as follow $f(x) = E[Y \mid X=x]$. Thats why I thought saying that outputting the mean is not a good idea was strange to me. As the data goes to infinity I guess we should output $f(x) = E[Y \mid X=x]$. Anyway, Is the quantity I suggested not the same as the mean you suggested? Or is your suggestion that $f(x) = E[Y]$ is a bad idea even if we knew the true distribution? $\endgroup$ – Charlie Parker Sep 5 '16 at 21:56
  • $\begingroup$ @CharlieParker if $x_{p,i} = \overline x_a$ for every $i$ that model is probably useless (unless the actuall values were, indeed, $x_{a,i} = \overline x_a$). $\endgroup$ – Firebug Sep 5 '16 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.