# How can a network with only ReLU nodes output negative values?

I'm trying to use an api with a feedforward neural network for time series forecasting. For dense aggregate data it works fine, but for sparse data it sometimes forecasts negative values, even though my historical data has only positive values.

The source code is very dense, and I might be missing a line or two, but as far as I can tell, the input layers and the hidden layers are all ReLU nodes.

Assuming I am correct, how can a network with only ReLU layers lead to negative values, especially if none of the training data has negative values?

• what about the output layer activations? – shimao Aug 17 '18 at 1:44
• also, might the api automatically normalize the predictors and targets, and unnormalize the network outputs? – shimao Aug 17 '18 at 1:46

$$f(x) = \max\{0, x\}$$