Truncated data in Neural Network regression I'm working on a Neural Network regression problem and the variable of interest (trade flows) that I'm trying to predict is skewed and truncated at 0.
I first tried to fit the model without prior transformation of the trade flows, only by scaling it in order to normalise the loss function. This lead to the network only predicting zeros. I'm using a ReLu activation in the output layer so as not to have negative values for the prediction.
After scaling the data by $log(x +1 )$ I got rid of the skewness, but I'm still left with 15% zeros.
Is there a way to somehow penalise the zeros? Might it be that I'm just not training long enough, that the network has thus not started to converge and I'm still underfitting?
My hyperparameters are set via a genetic algorithm. It might be that I need to crank up the upper bound of steps available for training.

 A: I'm speculating that the primary issue here is a good output layer and suitable loss function. In some sense your loss function should reflect your ultimate goal/business metric (or whatever else you really want to optimize).
Here are some options you could consider:

*

*One option is to have two outputs: a binary prediction (is this a 0 or a value >0) and a continuous prediction e.g. of log-transformed values. Your loss function could then be a weighted sum of the $\boldsymbol{w_1} \times$ binary log-loss + $\boldsymbol{w_2} \times$ mean squared error for non-zero records (or some other suitable loss function for the second part). That's what in statistics you would call a zero-inflated model. How you weight the two components of the loss function is the main challenge here and will depend on what truly matters to you in the end.

*If you think the underlying process is rather that the 0s are not truly zeros, but rather really small values that cannot be distinguished from zero, and if you additionally know that they all must be below some value $c$, then you could use a loss function that considers these values as censored below $c$ (this all also works if you have a specific censoring value for each record). In that case, you can use survival analysis techniques for censored observations.

*You could work with square-root transformed data as an easy to implement alternative to what you did. I doubt it will make too much of a difference, but could help.

*Does your data have some underlying data generation process that could be expressed as a count variable or binomial variable (e.g. is this the proportion of sales interacton for a customer that resulted in a sale or something similar - it would be especially useful, if you knew the denominator and could use it in predictions, in the example that would be how many sales interactions there were for a customer)? If so, there's some suitable statistical models your could use the likelihood from as a loss function (you would predict the parameters of the model for the record). E.g. zero inflated binomial (the output of the neural network would be two probabilities - or logits of probabilities - one for 0 or not zero probability and the second one for the probability >0), zero-inflated Poisson (output would be a logit-probability for being 0 and one log-rate) etc.

