3
$\begingroup$

My gradient boosting regression model (GBM) is trained to minimize mean absolute error (MAE) but gives the same prediction for every record on my highly skewed dataset. I believe there is a quick fix to the immediate problem (use RMSE) but my situation is complicated, and I worry that using RMSE will lead to a new set of problems that are much worse.

Background: My model must predict a continuous percentage -- not a discrete class -- and unfortunately about 85% of my records have a target / response value of 100%. Of the remaining 15%, about half the records have a target of 0% while the others have values somewhere between 0% and 100%. I suspect that my GBM is producing the same score for every record because the median target value at each tree's terminal nodes is 100%.

MAE vs RMSE: I want to minimize MAE because from my company's business perspective a prediction that is off by 10% is exactly twice as bad as a prediction that is off by 5%. Using RMSE instead of MAE fixes my identical predictions problem but heavily skews my predictions towards the outlier and produces many predictions in the range of 15%-20%, well below the market rate. My company would be forced out of business if it had to offer rates so low because no company would buy the product. On the other hand, my company would also be forced out of business if it always offered rates of 100%, since any outcome less than 100% would be a financial loss.

Regression vs Classification: I can't frame this as a classification problem because if I pick some arbitrary threshold (say, 60%) for binary classification, my company basically concedes the entire lower end of the market, since it wouldn't be able to differentiate a 40% customer from a 55% customer, since both would be lower than the example 60% threshold.

What should I do?

$\endgroup$
1
$\begingroup$

Please correct me if I am wrong, but it sounds like you are looking for a sweet spot between the sharpness of the L1 penalty and the smoothing of the L2 penalty. The elastic net might do the job.

Slide 9 of https://web.stanford.edu/~hastie/TALKS/enet_talk.pdf seems to be capturing the issue you mention

$\endgroup$
  • $\begingroup$ I checked out slide 9. I've used the elastic net, and I'm familiar with the L1 and L2 penalty, but I don't see how that relates to my particular problem. I didn't specify anything about the number of inputs I have, so variable selection and/or the penalization of correlated input variables doesn't seem relevant. Can you clarify? $\endgroup$ – Ryan Zotti Oct 12 '17 at 15:49
1
$\begingroup$

Probably, you can log transform (or any other scale transform) the target variable and then use RMSE. It might remove the high outlier impact.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.