I'm using the root mean squared error (RMSE) as a metric for tuning the parameters of my model in a regression problem through cross-validation. However, I'm not so much interested that all predictions are good, I want that about 20% or 40% percent of my predictions to be "spot-on" and don't care if the other 80% or 60% are garbage.
What metric would be best for this?
You could try the Huber loss function. It penalizes close predictions more than predictions that are very wrong.
Using the usual notation for Huber Less, define $a_i$ as your residual, $Y_i - \hat{Y}_{i}$. Huber loss is defined as
$$ L_\delta(a) = \begin{cases} \frac{1}{2} a^2, & \text{for $|a| < \delta$} \\ \delta * (|a| - \frac{1}{2}\delta), & \text{for $|a| \geq \delta$} \end{cases}$$
You define a threshold of "closeness", $\delta$. If your prediction is within this threshold, your loss is quadratic. If your prediction is outside of this threshold, your loss is linear. This let's the model prioritize fitting points that your model is close to getting correct and de-prioritize outliers. If you want to de-emphasize outliers even further, you could increase the polynomial degree in the case when $|a| < \delta$ and/or decrease the polynomial degree when $|a| \geq \delta$.
In R, you can calculate Huber Loss with the yardstick package. I'm not aware of any out-of-the-box implementations of Huber loss as a standalone metric in Python. Sklearn let's you fit linear regression with Huber Loss. but Huber Loss isn't one of sklearn's scoring metric.
Answering my own question here, it seems that the root mean squared log error (RMSLE) is a suitable metric, see this CV post
