Reduce over-forecasting in H2O regression with tree-based algorithms I'm trying to solve a time series forecasting problem. While accuracy is fine, I'm now trying to punish / reduce under-forecasting events where the forecast is too low.
Forecasting approach:


*

*Build lag features (e.g. truth from 1/2/3 ... /12 periods ago)

*Build trailing features (e.g. moving average over preceding X periods)

*Train H2O regression model (GBM) to predict truth based on features


Result:


*

*Forecast accuracy is good on average

*But in total, too many instances of under-forecasting


How to 'punish' under-forecasting more than over-forecasting when fitting the ML model?
I've tried log-transforming the target variable, this helped a bit but not enough. Are there other approaches I could use for this issue?
Thanks in advance for your comments!
 A: The first approach is to transform the target variable. You've already tried log-transforming. I've also tried the cubing/cube-root approach that @mkt mentions in the comments. I've had limited success in both cases - I suspect the machine learning model is seeing the different distribution and adapting to it.
Another way is to weight the records it predicted to be too low (this is described at the end of this answer).
But, what I would do, and have done, is see if you can create a model that will predict the likelihood of your model being too low or too high.  You can use the output from your current model, compared to the correct answer, to create some training data. Build a binomial model to predict its low/high tendency. (Or a 5-level or 6-level multinomial, with some thresholds, so you get: "very low", "low", "slightly low", "slightly high", etc.; another variation is a regression model to predict the actual error.)
Now take the output of that model and feed it in as an extra column and re-train your original model.
So, in summary:


*

*Make model A on all inputs in T, your training data, to predict target.

*Use model A to generate T2, which is like T but target is replaced by if the prediction was too high or too low.

*Make model B on T2.

*Add output of model B to T,  as an extra input column, to make T3.

*Make model C on all inputs in T3, to predict target.


(In production, the live data goes into model B, it adds another column, then the enhanced live data is fed into model C: model A is only used in training.)
Model C should be equal to or better than model A.
Now, maybe, this hasn't answered your question, because it gives equal weight to too high and too low? You can still combine it with the log/power idea. Or you can move the thresholds (with the multinomial categorization approach). But the thing I would try first is to weight the data: create 10 times more "too low" training records in T3 than "too high". This forces the model to try and do them better.
As you are using H2O, this comes built in: http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/weights_column.html
I.e. use h2o.ifelse() to add a "w" column, with 10 for too-low, 1 for too high. Then:
modelC <- h2o.gbm(..., weights_column = "w", ...)

(You could also do this weighting idea without making model B: make model A, see which direction it is wrong in, use that to add "w", and use that to remake model A... this is boosting, so kind of how GBM is already working; you are just taking control of telling it what is important.)
