1
$\begingroup$

I have the following data over time:

img

that means data collected for a single variable like CPU usage in lowest, highest, and average mode over time every 5 mins (data granularity = 5mins) like the following data frame:

|    | timestamp           |   min cpu |     max cpu |     avg cpu |
|---:|:--------------------|----------:|------------:|------------:|
|  0 | 2017-01-01 00:00:00 |    715147 | 2.2233e+06  | 1.22957e+06 |
|  1 | 2017-01-01 00:05:00 |    700474 | 2.21239e+06 | 1.21132e+06 |
|  2 | 2017-01-01 00:10:00 |    705954 | 2.21306e+06 | 1.20663e+06 |
|  3 | 2017-01-01 00:15:00 |    688383 | 2.18757e+06 | 1.19037e+06 |
|  4 | 2017-01-01 00:20:00 |    688277 | 2.18368e+06 | 1.18099e+06 |

I sliced the dataframe and worked on a univariate time-series data problem as follows:

|    | timestamp           |     avg cpu |
|---:|:--------------------|------------:|
|  0 | 2017-01-01 00:00:00 | 1.22957e+06 |
|  1 | 2017-01-01 00:05:00 | 1.21132e+06 |
|  2 | 2017-01-01 00:10:00 | 1.20663e+06 |
|  3 | 2017-01-01 00:15:00 | 1.19037e+06 |
|  4 | 2017-01-01 00:20:00 | 1.18099e+06 |

I split data and applied PI (Prediction Interval) using a regression:

|                     |        pred |   lower_bound |   upper_bound |
|:--------------------|------------:|--------------:|--------------:|
| 2017-01-25 00:00:00 | 1.15232e+06 |   1.12482e+06 |   1.1874e+06  |
| 2017-01-25 00:05:00 | 1.14453e+06 |   1.10052e+06 |   1.18994e+06 |
| 2017-01-25 00:10:00 | 1.14033e+06 |   1.08739e+06 |   1.20795e+06 |
| 2017-01-25 00:15:00 | 1.13669e+06 |   1.0843e+06  |   1.20252e+06 |
| 2017-01-25 00:20:00 | 1.1271e+06  |   1.06837e+06 |   1.19865e+06 |

img img


question:

Since I'm interested in upper_bound upper prediction limit only, (if you see this image from Jason for Relationship between prediction, actual value and prediction interval) then the frame of my problem changed from PI to the simple problem of target prediction. then:

1. Does it mean I can easily use normal target prediction metrics ($MSE$, $MAE$, $MAPE$, $R^2 score$ ) instead of PI metrics ($PICP$, $PINC$, $ACE$, $MPIW$, $PINAW$, $PINRW$, $score$ ) to evaluate the used regression/predictive model as they treat in package? ref.

# Prediction error
# ==============================================================================
from sklearn.metrics import mean_squared_error
error_mse = mean_squared_error(
                y_true = data_test['avg cpu'],
                y_pred = predictions.iloc[:, 0]
            )

print(f"Test error (MSE): {error_mse}")

In other words (regardless of the way is treated in package):

2. Is it fine in academics and papers that one treats the evaluation classic use of normal target prediction metrics ($MSE$, etc ) instead of PI metrics ($PICP$, etc. ) for evaluation if you are interested in only one target amongst upper\lower\targetin PI tasks? (It would be great if you cite an example paper)

Thanks in advance


Related materials found:

$\endgroup$
15
  • 2
    $\begingroup$ It is unclear what you're aiming for. Why are you only interested in the upper bound? Why do you produce interval forecasts if you are interested in a single value? $\endgroup$ Mar 9 at 7:21
  • $\begingroup$ @picky_porpoise, please point out which of Qs 1,2, or 3 are vague so that I can explain further. Why are you only interested in the upper bound? Due to the Upper bound gives us a better\fair measurement for our production. Why do you produce interval forecasts if you are interested in a single value? Apart from the fact that in data collection there are CPU Max, CPU Avg, CPU Min measurements based on domain knowledge received from domain experts we just consider avg CPU measurement for our PI task and focus on its Upper bound forecast for future CPU consumption recommendation sys. $\endgroup$
    – Mario
    Mar 10 at 12:24
  • $\begingroup$ The questions are not that vague, but without understanding the context they are hard to answer. Your comment mentions 'fair measurement' and 'domain knowledge', but this can mean many things. Can it be, that you are interested in a quantile, i.e. what level of CPU usage is not exceeded with a certain high probability? $\endgroup$ Mar 10 at 14:50
  • 2
    $\begingroup$ If you need distinct answers to multiple questions, they should be posted separately. $\endgroup$
    – Dave
    Mar 10 at 22:14
  • 1
    $\begingroup$ "Is it fine in academics and papers that one treats the evaluation classic use of normal target prediction metrics (MSE , etc ) instead of PI metrics (PICP , etc. ) for evaluation if you are interested in only one target amongst upper\lower\targetin PI tasks? " It is difficult to assess how fine it is. What is the background? Optimization of MSE is a decent goal. Sure, if the final goal is prediction interval coverage probability (PICP) then this might be potentially targeted more directly. But it is unclear what the situation is. Sometimes MSE can be a better goal to optimize than PICP. $\endgroup$ Mar 11 at 11:44

1 Answer 1

1
+50
$\begingroup$

Since I'm interested in upper_bound upper prediction limit only ... Does it mean I can easily use normal target prediction metrics ($MSE$, $MAE$, $MAPE$, $R^2 score$ ) instead of PI metrics ($PICP$, $PINC$, $ACE$, $MPIW$, $PINAW$, $PINRW$, $score$ ) to evaluate the used regression/predictive model

You can use those metrics (and especially for training as explained here: Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified?).

However to evaluate the models you might want to use a cost function that is closest to your target goal. If this is accuracy of the upper bound of a prediction interval, then use that as metric.

Is it fine in academics and papers that one treats the evaluation classic use of normal target prediction metrics (MSE , etc ) instead of PI metrics (PICP , etc. ) for evaluation if you are interested in only one target amongst upper\lower\target in PI tasks? (It would be great if you cite an example paper)

If the paper wants to treat a study about PI then it should be about PI in it's text and not about MSE.

$\endgroup$
1
  • $\begingroup$ Do you have any input about this post by any chance? $\endgroup$
    – Mario
    Mar 17 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.