# Regression - extremely skewed response with a large, sparse matrix of boolean predictors

I'm working with a dataset that contains:

• $y$: the response variable that is 98% zero, but in the remaining 2% of cases it has extremely skewed real values (not integers), ranging from sub 1 to over 100,000.
• $X$: a sparse matrix of approximately 1000 variables containing 0s and 1s
• $w$: a weight variable that is desired to be used in the error calculation

I've tried the following approaches:

• Tweedie regression with xgboost using linear booster -- tried it with both the original $y$ variable and $log(y+1)$, found that transformation actually makes it worse
• Binning $y$ into a set of buckets, then using a multi-class classification (also with xgboost) and then to convert the predictions back into original scale, for each observation I took the weighted average of mean $y$ values in each bucket (from the train set), with weights being the inferred class probabilities -- it did not perform as well as the regression approach

Other ideas that I have:

• An ensemble of (1) a classifier that discriminates between zero and non-zero values and (2) a regression like the one described above
• A neural network

So far, I've found that all approaches return predictions that do not follow the same distribution as the $y$, my predictions have many more non-zero values than the actual response. Given the description of the problem, are there any modelling techniques that you could recommend? Ideally, the solution would (1) be computationally efficient as the dataset is quite large, and (2) would accommodate for supplementary weights.

Many thanks!

EDIT:

I've tried an ensemble of (1) a classifier that determines whether or not the value is expected to be zero and (2) regression -- the result was disappointing.

I've also tried fitting a neural network, but that too struggles to find any meaningful relationships in the data, so far the aforementioned Tweedie regression is the best. I've attached the neural net architecture, if you guys have any suggestions on how to improve it, please let me know.

• The resonse values that get passed to the NN are $log(y+1)$
• On the "predictions are non-zero" part, they should be! Remember that in prediction, you are estimating the conditional expectation of $y$. As long as there is a non zero probability that y is non-zero, the prediction will not be exactly zero. This is an expression of regression to the mean, and is a feature of all models. – Matthew Drury Feb 5 '18 at 22:27
• Thank you, Matthew! Yes, I understand why they are not zero due to regression to the mean, however, given the skewness of $y$, I thought that frequent non-zero predictions are an indicator that using the methods I outlined above is simply not appropriate here. Any thoughts on that? – de1pher Feb 5 '18 at 22:31
• When you run your regressions on the transformed $y$ variable, are you simply running the predictions through the inverse transform, or are you accounting for the fact that, e.g., $\text{E}\exp(y) \neq \exp(\text{E}y)$? – jbowman Feb 5 '18 at 22:42
• @jbowman, good question, after generating predictions, I back-transform the results using $exp(\hat{y})-1$. which restores the original scales. Do you have any suggestions on that? – de1pher Feb 5 '18 at 22:54
• I am guessing your back-transformed predictions are biased quite low. Whether or not there's much you can do about it is another thing! The canonical back-transform is $\exp(\hat{y} + \hat{\sigma}^2/2)$, where $\hat{\sigma}^2/2$ is the (estimated) variance of $\hat{y}-y$ (on the log scale, be it noted.) But of course that requires $\hat{\sigma}^2/2$ , which may be estimated with such poor accuracy that the end result of the back-transform remains worse than not doing the transform at all. – jbowman Feb 5 '18 at 22:58