1
$\begingroup$

My case, as it seems to me, should be quite common, yet I cannot find any information.

The situation is as follows: there is a regression model, and for each predicted value, there are multiple true values. For example:

enter image description here

What methods are there to evaluate the predictions?

Of course, I can evaluate predictions relative to the median value or the mean.

For example, AE would be: $$ AE_i = \mid\hat{y_i} - \frac{1}{3}({y_i}_1 + {y_i}_2 + {y_i}_3)\mid = \mid\hat{y_i} - \bar{y_i}\mid $$

where $\hat{y_i}$ is the i-th prediction, $\bar{y_i}$ is the mean of true values ${y_i}_1, {y_i}_2, {y_i}_3$.

However, errors relative to the mean or median do not reflect the variance of the true values. My idea is that if the prediction falls inside the range of the true values, the error should be significantly smaller compared to the error relative to the mean, or even equal to zero.

I can think of several metrics.

  1. Percent of predictions falling inside the corresponding true value range. This metric, however, does not reflect how bad are predictions which do not fall inside the ranges.
  2. "Relative absolute error". The sum of distances between the prediction and real values devided by the sum of distances between the mean and the real values. $$ e_i = \frac{\sum_j{\mid \hat{y_i} - {y_i}_j \mid}}{\sum_j{\mid{y_i}_j - \bar{y_i} \mid}} $$ where $\bar{y_i}$ is the mean of real values ${y_i}_1, {y_i}_2, {y_i}_3$.
  3. Absolute error relative to quantiles. For example, to evaluate predictions relative to quantiles 0.1 and 0.9:

$$ E = {Q_i}_{0.1} - \hat{y_i}, \quad \text{if }\enspace \hat{y_i} < {Q_i}_{0.1} \\ E = 0, \quad \text{if }\enspace {Q_i}_{0.1} \le \hat{y_i} \le {Q_i}_{0.9} \\ E = \hat{y_i} - {Q_i}_{0.9} , \quad \text{if }\enspace {Q_i}_{0.9} < \hat{y_i} $$

where ${Q_i}_{0.1}$ and ${Q_i}_{0.9}$ are the corresponding quantiles of the true values ${y_i}_1, {y_i}_2, {y_i}_3, ...$

In my case, I have 10 to 90 true values for each prediction. Their distribution is not symmetrical (otherwise, errors relative to the median could be compared to the spread or range width of the true values).

Do you think any of the errors 1.-3. could be useful? What are the more common ways to evaluate predictions in such cases?

Some specifics of my case. I want to predict the time of computation of a piece of code. The computation time naturally fluctuates, however, mainly in the direction of larger values (longer computation times). It also has outliers. Since the code execution times vary, I make multiple measurements. Execution time depends on the hardware and some other parameters. I conduct similar sets of measurements on several machines, also for different data sizes (varying one parameter that changes the size of the data used in computations and impacts execution time) and for different code pieces (defined by a large set of parameters). Thus, for each combination of a machine, a particular piece of code and a data size, I have multiple execution time measurements. For different machines, codes and data sizes, the execution times magnitude and spread (variance) are also different. To mitigate uncertainty, I conduct more measurements when the execution time variation is large.

I want to build a prediction model, that for the given machine, piece of code and data size, would predict the execution time. To evaluate my model predictions, I want a metric that would not only tell how far/close to the median time the predictions are, but also consider the spread of the real values (measured execution times) because if the spread is large, predictions farther away from the median still can be considered as good predictions.

$\endgroup$
13
  • $\begingroup$ Where do the multiple true values come from? Can you treat them as separate observations? $\endgroup$ Jul 15, 2023 at 13:18
  • $\begingroup$ Yes, they are separate observations. $\endgroup$
    – Peter
    Jul 15, 2023 at 16:24
  • $\begingroup$ So is it like a repeated measures ANOVA where you have three observations for Peter, three for Paul, and three for Mary? $\endgroup$
    – Dave
    Jul 15, 2023 at 16:30
  • 1
    $\begingroup$ Please give more detail by editing the question to elaborate on that. Depending on those details, I have several ideas, as will others. $\endgroup$
    – Dave
    Jul 15, 2023 at 18:41
  • 1
    $\begingroup$ The key point (that I will keep emphasizing) is how you wound up with multiple measurements. Depending on that answer, I can think of several ways to handle this, all of which might be inappropriate for your particular situation, so please flesh out the details. $\endgroup$
    – Dave
    Jul 16, 2023 at 0:41

1 Answer 1

0
$\begingroup$

If you have only point predictions, choose a point forecast error measure that elicits the functional you are looking for, e.g., the MSE for the mean, or the MAE for the median, or a pinball loss for a quantile prediction. See here.

Interval predictions can be assessed using interval scores.

Full predictive densities can be evaluated using proper scoring rules. More information can be found here.

However, if you use your predictions for some specific subsequent decision, the approach of using the best forecast (chosen by one of the error measures above) and then optimizing the decision conditional on the forecast is not guaranteed to yield the best final decision. If so, it might be best to directly assess a decision you make on your data.

$\endgroup$
11
  • $\begingroup$ Thank you for your informative answer. Unfortunately, I do not see how this can help in my case. I have a point forecast that I want to evaluate with respect to multiple true values. I want metrics that would reflect the distribution of real values. $\endgroup$
    – Peter
    Jul 20, 2023 at 1:24
  • $\begingroup$ That is my first paragraph. You will need to decide what a "good" point forecast would be. Do you want a point forecast that is incentivized to be the median of the observations? Use the MAE. Should it be the mean of the observations? Use the MSE. Should it be a specific quantile? Use a pinball loss. If you specifically want to address the spread of your outcomes, quantile predictions may be appropriate, or even multiple ones, i.e., an interval forecast. $\endgroup$ Jul 20, 2023 at 7:17
  • $\begingroup$ Thanks again! I already have errors with respect to the median. Changing my model is not an option. Thus, I need a metric to evaluate point predictions that reflect the distribution of real values. $\endgroup$
    – Peter
    Jul 20, 2023 at 12:34
  • 1
    $\begingroup$ OK, that helps. It does sound like you will need to experiment a bit with potential error measures to find out which one would be most valid. I would have recommended dividing the AE by some measure of dispersion (difference between max and min, some inter-quantile range, standard deviation), or a function of that, but it seems from your last comment that you don't want that. Maybe use a function of $q_{1-\alpha}-\hat{y}$ (the difference between a high quantile and the prediction) and $\hat{y}-q_\alpha$ (the difference between the prediction and a low quantile) to capture asymmetry? $\endgroup$ Jul 21, 2023 at 5:58
  • 1
    $\begingroup$ That is certainly a possibility, if it works with what you want to use the predictions for. (I understand you can't change your algorithm, but this does suggest to me to model two quantiles, and perhaps afterwards take the midpoint of the two predicted quantiles.) $\endgroup$ Jul 22, 2023 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.