# Dealing with excessive number of zeros

ipdb> np.count_nonzero(test==0) / len(ytrue) * 100
76.44815766923736


I have a datafile counting 24000 prices where I use them for a time series forecasting problem. Instead of trying predicting the price, I tried to predict log-return, i.e. $$\log(\frac{P_t}{P_{t-1}})$$. I have applied the log-return over the prices as well as all the features. The prediction are not bad, but the trend tend to predict zero. As you can see above, ~76% of the data are zeros.

Now the idea is probably to "look up for a zero-inflated estimator : first predict whether it's gonna be a zero; if not, predict the value".

In details, what is the perfect way to deal with excessive number of zeros? How zero-inflated estimator can help me with that? Be aware originally I am not probabilist.

P.S. I am working trying to predict the log-return where the units are "seconds" for High-Frequency Trading study. Be aware that it is a regression problem (not a classification problem).

Update

That picture is probably the best prediction I have on the log-return, i.e $$\log(\frac{P_t}{P_{t-1}})$$. Although it is not bad, the remaining predictions tend to predict zero. As you can see in the above question, there is too many zeros. I have probably the same problem inside the features as I take the log-return on the features as well, i.e. if $$F$$ is a particular feature, then I apply $$\log(\frac{F_t}{F_{t-1}})$$.

Here is a one day data, log_return_with_features.pkl, with shape (23369, 30, 161). Sorry, but I cannot tell what are the features. As I apply $$\log(\frac{F_t}{F_{t-1}})$$ on all the features and on the target (i.e. the price), then be aware I added 1e-8 to all the features before applying the log-return operation to avoid division by 0.

• There is no perfect way.....the sooner you accept this the better Commented Nov 14, 2018 at 0:13

There is no perfect solution to anything when modelling. A good method for handling zeros will depend on the application you are working on. I am not a neural network expert, nor am I a time series expert, so I can't help you without knowing a little more about the problem you are working on. However, I do know a little something about zero inflated models/hurdle models.

A hurdle model posits that the zeros and non-zeros occur from different processes. Here is an example of zeros occurring from a hurdle process:

Flip a coin. If the coin is heads, then roll a die and record the outcome. If the coin is tails, record 0.

In order to get a non-zero result you have to "jump the hurdle" (i.e. the coin has to be heads). The processes which result in zeros and non-zeros are distinct.

Here is an example of a zero inflated process:

Flip a coin. if the coin is heads, flip another coin. If the second coin is heads, record 1. If the second coin is tails, record 0. If the first coin is tails, record 0 and do not flip the second coin.

The zeros in this process are a mix of the first and second process. You can see where the name "zero inflation" comes from.

Now, back to your problem. I would determine which of these, hurdle or zero inflation, is most realistic. You know more about the process than I do, so I can't make any recommendations. Once you have decided, you'll be able to make more informed choices about your modelling.

How can this help you? Suppose that your process is best modelled as a hurdle process. Maybe the returns are zero only when it rains. Then, you can model the hurdle process as a function of the covariates, and use the result of that process to model the returns.

• I added new information in the question. Maybe it would help. Commented Nov 14, 2018 at 16:43

A simplistic approach would be to calculate the frequencies of all the possible values and to compare the ratios between frequencies. For example, to compare the frequency of getting a value of 1 against a higher value, sum the frequency of all the higher values to use as the divisor of the value 1 frequency. Simarly for negative values and mixtures of positive and negative values. As you have a very large sample size, this should be quite reliable.

• Your approach seems interesting, but I am a bit confused. Are you up to build a detailed example inside your answer? Commented Nov 15, 2018 at 13:24
• Btw, it is not a classification problem, but a regression problem. I think that technique won't work cause the target is continuous. +1 for the try. Commented Nov 16, 2018 at 12:25
• I think my answer was too simplistic, you could calculate the standard deviation of the non-zero values and construct confidence intervals accordingly, e.g. 95% would be 1.96 standard deviations. Commented Nov 16, 2018 at 12:27

You could modify your cost function to take into account the imbalance between categories. In other words, make zeros count less. Here is an example using binary crossentropy in keras. The parameter 'weight' in this case is the fraction of 1 s.

import keras.backend as K

def make_loss(weight):

def loss_function(y_true, y_pred):
return -K.mean(y_true * K.log(y_pred) + weight * (1 - y_true) * K.log(1 - y_pred))

return loss_function

• It is not a classification problem, but a regression problem. I think that technique won't work cause the target is continuous. +1 for the try. Commented Nov 16, 2018 at 12:26