I am trying to predict the revenue that a click generates on an average (revenue per click). There are K different items that get shown to different users. My training data consist of clicks. A fraction (say 10%) of those clicks lead to conversion and generate a positive revenue.

My target variable is the revenue amount in dollars. As such, this is a regression problem. I am using LightGBM (but have also used XGBoost earlier) with Tweedie regression (since revenue is non-negative and has a huge peak at 0).

By and large, I get good predictions (predicted revenue is close to actual revenue in holdout, as measured by RMSE) for most of the K items. However, there are certain items for which the model is severely underpredicting (50% or more).

All those items belong to two different categories (think of categories as "clothing" vs "kitchen appliances" on Amazon). For other categories, the predictions are good.

My suspicion is that my feature set does not have good enough features for those categories.

When I look at feature importances, the features for those two categories do not show up in the top 15 features (the topmost feature is "itemId" which is expected). So, I decided to build a model just over data for one of the problematic category. However, the performance of that model (RMSE) was worse than the unified model (for all categories).

It is certainly not an issue between training and holdout data skew since I have tried both in-sample and out-of-sample holdout. I tried in-sample holdout multiple times but getting the same under-prediction.

Any suggestions on what else I can try to improve the performance?

  • $\begingroup$ By underpredicting you mean that model is predicting values that are consistently lower than the true value? $\endgroup$
    – Ryan Volpi
    Jul 17, 2020 at 6:46
  • $\begingroup$ Yes. That is the case. I also figured out that while validation RMSE is quite close to training RMSE, holdout RMSE is way worse. So, the model must be overfitting. Trying to address that right now. If you have any suggestions, please let me know. $\endgroup$
    – Nik
    Jul 17, 2020 at 21:08
  • $\begingroup$ I did realize one thing that I didn't tune the tweedie_variance_power (which defaults to 1.5). I did try tuning it but it makes no difference in the validation RMSE (it is as if the model doesn't even recognize the parameter. May be I am not specifying it properly?). $\endgroup$
    – Nik
    Jul 17, 2020 at 21:10
  • $\begingroup$ Hi @Nik, did you found some ways to improve your performant and fix the problem you described ? $\endgroup$ Jan 14, 2021 at 2:20
  • 1
    $\begingroup$ @demarsylvain I solved the issue by splitting revenue per click into click to conversion and revenue per conversion funnels. Click to conversion was a classifier that emitted the probability of conversion, given that the user clicked. Revenue per conversion was fairly consistent. The issues were mostly with Click to conversion which I fixed by using Tensorflow model instead of Xgboost. $\endgroup$
    – Nik
    Mar 18, 2021 at 22:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.