I'm working with a dataset that contains:
- $y$: the response variable that is 98% zero, but in the remaining 2% of cases it has extremely skewed real values (not integers), ranging from sub 1 to over 100,000.
- $X$: a sparse matrix of approximately 1000 variables containing 0s and 1s
- $w$: a weight variable that is desired to be used in the error calculation
I've tried the following approaches:
- Tweedie regression with
xgboostusing linear booster -- tried it with both the original $y$ variable and $log(y+1)$, found that transformation actually makes it worse
- Binning $y$ into a set of buckets, then using a multi-class classification (also with
xgboost) and then to convert the predictions back into original scale, for each observation I took the weighted average of mean $y$ values in each bucket (from the train set), with weights being the inferred class probabilities -- it did not perform as well as the regression approach
Other ideas that I have:
- An ensemble of (1) a classifier that discriminates between zero and non-zero values and (2) a regression like the one described above
- A neural network
So far, I've found that all approaches return predictions that do not follow the same distribution as the $y$, my predictions have many more non-zero values than the actual response. Given the description of the problem, are there any modelling techniques that you could recommend? Ideally, the solution would (1) be computationally efficient as the dataset is quite large, and (2) would accommodate for supplementary weights.
I've tried an ensemble of (1) a classifier that determines whether or not the value is expected to be zero and (2) regression -- the result was disappointing.
I've also tried fitting a neural network, but that too struggles to find any meaningful relationships in the data, so far the aforementioned Tweedie regression is the best. I've attached the neural net architecture, if you guys have any suggestions on how to improve it, please let me know.
Key notes about the NN:
- It has 2 inputs, an indicator matrix and time series of the response
- The resonse values that get passed to the NN are $log(y+1)$
- I'm using a sigmoid activation on the indicator matrix, no activation on the LSTM layer and ReLU activation on the intermediate dense layers -- the final layer has not activation function