# Best practice for normalizing output in regression

I am working on some complicated regression problems that I am fitting with deep neural networks. In other to make these deep networks trainable, there are normalisation steps all over the place in my networks. The output in natural units is of course not normalised.

What I have been doing so far is adding a zero-bias simple multiplicative neuron at the end of each regression output, so that the network can learn an appropriate 'denormalisation' itself. However, I just realised this has a quite adverse effect on the speed of convergence of my network; likely due to the strong coupling of this neuron with all other unknowns, thus creating a very poorly conditioned 'valley' for gradient decent to contend with.

Initialising the weight of this denormalising neuron with for example the mean of the output vector helps a lot in further training; cutting the required iterations to get to the same loss by an order of magnitude. Yet I am worried I am still adversely influencing training despite the initialisation; even with good initialisation the condition number will likely suffer. Thats what happens when you add depth to a network in general, but do I have to waste my depth on this?

The obvious alternative is to do this kind of normalisation outside of my network. But then the ratio is non-trainable and might still 'clash' with the normalisation steps in the network. And I don't like having another little bit of state to keep track of in my model; its a single weight linear transform; my fancy computational graph framework should be able to handle it, right?

I feel like I am reinventing the wheel here. Searching the internet for best practices for this problem does not yield much. Anything you can recommend?

EDIT:

Having experimented with the options more, separately training the denormalisation layer from the rest of the network appears to be the most effective approach. That is, do separate training passes where only either the denormalisation layer or the rest of the network is trainable. However, it really uglifies my code, so I am not too happy with it. Overall the dynamic of such a denormalisation layer is really the same as a BatchNorm layer; it has its own micro-loss (match normal-data to non-normal data or vice-versa) that it optimizes for independently of the rest of the networks objectives by adjusting its internal state. However a denormalisation layer can only update during backprop since it needs to recieve the downstream signal to know how to update; so the implementation is hardly a copy-paste of a batchnorm layer; but I am hoping to create a clean and reusable component along these lines. But if anyone has suggestions along these lines they are very welcome too!

• It would help to know the structure of the neural network (how many input nodes, how many hidden layers and nodes, and how many output nodes?). Also, what kind of range would you like the output nodes to map to (i.e. [0,1])? Are you standardizing outputs to fit within some sort of range, or are you trying to speed up training by doing so? – rpatel Jul 10 at 10:53
• I have some suggestions, but would like to know a little bit more first – rpatel Jul 10 at 10:55
• The relevant part about the structure of the network is that it uses batch normalisation extensively. It can be single or multi output, and single or multi input. I dont want my output nodes to map to anything; my concern is matching any possible output range. Its 0-10000 for my current problem typically, but i'm trying to find a general solution, of matching this to batch-normalised signals. – Eelco Hoogendoorn Jul 10 at 11:06
• The wording of the question is still a bit confusing - are you asking how to bound the output/s of your neural network to any particular output range? – rpatel Jul 10 at 13:24
• I am not sure what you mean by 'bound'. The outputs of my neural network should match the training data. Either I normalise my training data outside the network, or I have at least one last layer in the network that is not batch normalised and has non-saturating activation. I outline the drawbacks of both approaches in my original post; I am wondering if there are other options I am missing. – Eelco Hoogendoorn Jul 10 at 14:32

All of the learning happens in the hidden layers. By the time we get to the output layer, we don't need to do any more work - there are already plenty of features on the second-to-last layer that have very high mutual information with the output. All we need to do in the output layer is a little book-keeping - averaging together features and centering and scaling them so match the response variable $y$. And the usual 1-node linear response using bias output layer is perfect for that.