If I understand this approach correctly, for given prediction
y_hat the following is true:
- With a chance of 75%, the value of
y_hatis lower than the actual value of
- With a chance of 25%, the value of
y_hatis higher than the actual value of
In order to evaluate this model, intuitively I would would compare the overestimations in the test to the underestimations. An "ideal" model would "underestimate" in 25% of the cases (i.e. the value of
y_hat is higher than the actual value of
Is this a valid approach?
Because I find it somehow confusing to not evaluate regression metrics like MAE, RMSE, Rsquared, etc. But these metrics do no seem suitable here (because we deliberately overestimate). Is there some asymmetric evaluation metric I am missing for this case? Happy if anyone can help clarify my confusion.
Note: I am aware of this post and the respective papers linked there. But is there a corresponding (python) implementation?