I am trying to predict the revenue that a click generates on an average (revenue per click). There are K different items that get shown to different users. My training data consist of clicks. A fraction (say 10%) of those clicks lead to conversion and generate a positive revenue.
My target variable is the revenue amount in dollars. As such, this is a regression problem. I am using LightGBM (but have also used XGBoost earlier) with Tweedie regression (since revenue is non-negative and has a huge peak at 0).
By and large, I get good predictions (predicted revenue is close to actual revenue in holdout, as measured by RMSE) for most of the K items. However, there are certain items for which the model is severely underpredicting (50% or more).
All those items belong to two different categories (think of categories as "clothing" vs "kitchen appliances" on Amazon). For other categories, the predictions are good.
My suspicion is that my feature set does not have good enough features for those categories.
When I look at feature importances, the features for those two categories do not show up in the top 15 features (the topmost feature is "itemId" which is expected). So, I decided to build a model just over data for one of the problematic category. However, the performance of that model (RMSE) was worse than the unified model (for all categories).
It is certainly not an issue between training and holdout data skew since I have tried both in-sample and out-of-sample holdout. I tried in-sample holdout multiple times but getting the same under-prediction.
Any suggestions on what else I can try to improve the performance?
tweedie_variance_power(which defaults to 1.5). I did try tuning it but it makes no difference in the validation RMSE (it is as if the model doesn't even recognize the parameter. May be I am not specifying it properly?). $\endgroup$