1
$\begingroup$

Say I'm building some kind of regression and I want to measure how good it is.

Of course I want to measure the absolute error between the prediction and the real outcome, and if that was all I cared about, I could use Mean Squared Error (MSE), but my regression also gives me information about the standard deviation expected from its predictions at each point and I would like to incorporate this standard deviation into the "goodness" of my regression.

Specifically, what I want is to penalize the errors more when the model estimates a small standard deviation about its predictions, but also penalize large standard deviations per se.

Is the likelihood all I need? I know that maximizing the log likelihood, for example, already takes into account the standard deviation, but I'm not sure that it takes into account the absolute error, or at least not explicitly.

Improve this question
$\endgroup$
2
  • $\begingroup$ What you mean when you say your regression gives you info on standard deviation? Also, maximization of the log likelihood considers standard deviation because it's a parameter of your data distribution, what you're trying to model, it's not like you're choosing what to use for optimizing it; it's like that by definitiom of what a likelihood is. I guess you might be mixing things a bit, unless you can make it clear. $\endgroup$
    – Lucas Farias
    Commented Apr 28, 2018 at 5:59
  • $\begingroup$ The input data has an standard deviation, but I'm referring to the standard deviation of the prediction. Take for example a gaussian process, it does not only predicts a point, but it also gives you an expected error (an standard deviation for that prediction). Think of it as a measure of uncertainty about the prediction. $\endgroup$
    – Lay González
    Commented Apr 28, 2018 at 6:02

1 Answer 1

Reset to default
4
$\begingroup$

You could construct a scaled error $s.e.$ by weighting the raw error inversely proportionally to its predicted standard deviation,
$$ s.e.:=\frac{y-\hat y}{\hat\sigma}, $$ where $y$ is the actual realization, $\hat y$ is the point forecast, and $\hat\sigma$ is the predicted standard deviation. Then take the mean of absolute values or mean of squares of $s.e.$s over the test sample. That would be the same as MAE and MSE but on scaled errors instead of raw ones. I would not be surprised to find out that such a measure of forecast accuracy already exists and has a name.

(Note that the established mean absolute scaled error - MASE - uses a different scaling than I am proposing here.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yeah, this is the obvious approach, but I don't know neither the name of it nor any other, so I'll take this as the answer unless somebody else comes with a better one. I'll just add that it reminds me about the Sharpe ratio both in form and in function. Thanks. $\endgroup$
    – Lay González
    Commented Apr 28, 2018 at 8:37
  • $\begingroup$ Although... this has the property that if sigma goes to infinity, the the error approached zero. I need to penalize large sigmas. $\endgroup$
    – Lay González
    Commented Apr 28, 2018 at 8:45
  • 1
    $\begingroup$ Yes, it reminds of the Sharpe ratio indeed. For distributions with infinite $\sigma$, you may use some lower-order moments such as $\mathbb{E}(|y|)$ in the denominator. (I included $\sigma$ as you had it in your question.) $\endgroup$
    – Richard Hardy
    Commented Apr 28, 2018 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.