1
$\begingroup$

First off, apologies for any malformed parts of this question. I'm a very much a newbie and still very much learning. I've tried to do as much research as possible before asking here.

I'm building/tuning a model to solve a tabular regression problem and predict a price for a widget. Some notable properties of the data/business domain:

  • The tabular data has a wide range of possible prices - from 10 dollars to 100,000+ dollars (a very small number are $1m+).
  • The model is being built to predict optimal purchase price - so in this case, under-prediction is much preferred to over-prediction (paying too little is good - paying too much is bad).

I'm building this using ludwig (and may experiment with H20 or scikit-learn) but the key thing I have control over now is the loss function.

I've built the first version of this model using RMSLE (Root Mean Squared Logarithmic Error) as the loss function and achieved decent results. I chose RMSLE because it optimizes for ratio-based losses -- a prediction of 100 with actual price 110 is the same as a prediction of 10,000 with an actual price of 11,000, which fits my problem space.

However an issue I've uncovered with RMSLE are that it penalizes under-prediction more (which is the opposite of what I want) and tends to be more forgiving of large error. The curve looks like:

enter image description here

I would like to build a loss function whose properties are similar to RMSLE in that it's ratio-based and can handle both large and small predicted/actual prices, but with a curve that's more asymmetric and penalizes overprediction more than underprediction. I read a bit about Linear Exponental Loss and the curve looks right:

enter image description here

But I'm concerned it might not share the properties of being ratio-based like RMSLE does.

So I seek your collective wisdom for advice and guiance. Many many thanks in advance and again, apologies for any foolish/malformed portions of this question.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Welcome to Cross Validated! Do you have some sense of by how much overpredictions are worse than underpredictions? For instance, predicting $5$ dollars too low is equivalent to predicting $X$ dollars too high. Is $X=1?$ Is $X=4.99?$ $\endgroup$
    – Dave
    Commented Jul 13, 2022 at 16:58
  • 1
    $\begingroup$ maybe stats.stackexchange.com/questions/255652/…? $\endgroup$
    – Alberto
    Commented Jul 13, 2022 at 18:01
  • $\begingroup$ Thank you for the welcome @Dave. I don't yet have an exact answer. But I can tell you more about the business problem and perhaps reason it out: Users of the model are aiming to purchase for resale and make a profit. The cost of over-predicting is a potential loss -- if the actual price is 100 and prediction is 110, you would lose 10. If the actual price is 100 and prediction is 90, you would lose 0 (and potentially profit 10). So the business cost of predicting too low is a missed opportunity, and the business cost of predicting too high is an actual loss. Hope this is helpful. $\endgroup$
    – nlh
    Commented Jul 13, 2022 at 18:56
  • 1
    $\begingroup$ Thanks @AlbertoSinigaglia. The linked answer, suggests a piecewise loss function based on what I think is MSE? (or just SE?). I'd lose the ratio-based approach of RMSLE. Is there a way to create "piecewise" RMSLE? $\endgroup$
    – nlh
    Commented Jul 13, 2022 at 19:12
  • 1
    $\begingroup$ @nlh you can use the MSE that I've posted in the answer below (and add in front of that a coefficient with different based on the case x>0 or x<0) $\endgroup$
    – Alberto
    Commented Jul 13, 2022 at 20:56

1 Answer 1

Reset to default
1
$\begingroup$

Supposing that your predictions are always different from zero, since you want to be invariant to the scale, you can optimize the ratio: $$ loss(out, target) = |\frac{pred - target}{target}| $$ or if you want it to be differentiable, you can consider the square: $$ loss(out, target) = \left(\frac{pred - target}{target}\right)^2 $$

Now, those loss are symmetric, therefore overshooting or undershooting is equally penalized

If you have targets very close to 0, you might want to add a coefficient on the bottom to avoid division by zero, and their relative inaccuracies

Improve this answer
$\endgroup$
2
  • $\begingroup$ Isn't this the opposite of the asymmetric loss the question is looking for? $\endgroup$
    – whuber ♦
    Commented Jul 13, 2022 at 17:29
  • $\begingroup$ @whuber yes, however he already have a solution for the "unbalanced" using a system of equation, he just needed a scale invariant loss $\endgroup$
    – Alberto
    Commented Jul 14, 2022 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.