Is there an error metric that decreases the weight when the target is near zero?

Question

As precipitation prediction models can only predict positive values, they won't be able to undershoot small values by much. When it comes to overshooting, there is no boundary. High precipitation values can essentially be overshot and undershot equally, except a model predicts ridiculously large amounts. Furthermore, if previous weather has been dry, simple models, such as the moving average can easily predict zero values. This issue I'd like to address. I've come up with a custom variant of the RMSE (cRMSE). Would this address this issue?

np.sqrt(np.mean((y_true - y_pred)**2 + w * np.exp(-np.abs(y_true))))

The cRMSE is a custom implementation of the Root Mean Squared Error (RMSE) error metric. This could be a potentially useful approach for precipitation forecasting, as it incorporates an additional weighting factor w $\in\{ℝ|0<w<1\}$ applied to values close to zero for y_true.

The cRMSE metric could be useful in cases where you want to give less weight to values close to zero, for example, in situations where predicting zero values accurately is considered less important than predicting non-zero values. The weighting factor w allows adjusting the impact of the additional term in the error metric, and you can experiment with different values of w to find the best balance between accuracy for non-zero values and tolerance for zero values.

Answer 1

It sounds like you want to implement weighted RMSE, where the weights are set according to the y_true values. To do this in a way that will be useful for discriminating between models that make different predictions, you should use multiplication instead of addition on the weight term:

np.sqrt(np.mean((y_true - y_pred)^2 * w * (y_true))

The use of addition as you suggested would made the added error term dependent solely on the y_true values but not on the predictions - every model would see its error measure increase by a fixed amount, so it's the equivalent of simply adding an arbitrary constant. By using multiplication instead, the actual error terms get weighted by the y_true values, meaning that models which make different predictions will see their error measure change by different amounts.