-3
$\begingroup$

I am working on some complicated regression problems that I am fitting with deep neural networks. In other to make these deep networks trainable, there are normalisation steps all over the place in my networks. The output in natural units is of course not normalised.

What I have been doing so far is adding a zero-bias simple multiplicative neuron at the end of each regression output, so that the network can learn an appropriate 'denormalisation' itself. However, I just realised this has a quite adverse effect on the speed of convergence of my network; likely due to the strong coupling of this neuron with all other unknowns, thus creating a very poorly conditioned 'valley' for gradient decent to contend with.

Initialising the weight of this denormalising neuron with for example the mean of the output vector helps a lot in further training; cutting the required iterations to get to the same loss by an order of magnitude. Yet I am worried I am still adversely influencing training despite the initialisation; even with good initialisation the condition number will likely suffer. Thats what happens when you add depth to a network in general, but do I have to waste my depth on this?

The obvious alternative is to do this kind of normalisation outside of my network. But then the ratio is non-trainable and might still 'clash' with the normalisation steps in the network. And I don't like having another little bit of state to keep track of in my model; its a single weight linear transform; my fancy computational graph framework should be able to handle it, right?

I feel like I am reinventing the wheel here. Searching the internet for best practices for this problem does not yield much. Anything you can recommend?

EDIT:

Having experimented with the options more, separately training the denormalisation layer from the rest of the network appears to be the most effective approach. That is, do separate training passes where only either the denormalisation layer or the rest of the network is trainable. However, it really uglifies my code, so I am not too happy with it. Overall the dynamic of such a denormalisation layer is really the same as a BatchNorm layer; it has its own micro-loss (match normal-data to non-normal data or vice-versa) that it optimizes for independently of the rest of the networks objectives by adjusting its internal state. However a denormalisation layer can only update during backprop since it needs to recieve the downstream signal to know how to update; so the implementation is hardly a copy-paste of a batchnorm layer; but I am hoping to create a clean and reusable component along these lines. But if anyone has suggestions along these lines they are very welcome too!

Improve this question
$\endgroup$
5
  • $\begingroup$ It would help to know the structure of the neural network (how many input nodes, how many hidden layers and nodes, and how many output nodes?). Also, what kind of range would you like the output nodes to map to (i.e. [0,1])? Are you standardizing outputs to fit within some sort of range, or are you trying to speed up training by doing so? $\endgroup$
    – rpatel
    Commented Jul 10, 2018 at 10:53
  • $\begingroup$ I have some suggestions, but would like to know a little bit more first $\endgroup$
    – rpatel
    Commented Jul 10, 2018 at 10:55
  • $\begingroup$ The relevant part about the structure of the network is that it uses batch normalisation extensively. It can be single or multi output, and single or multi input. I dont want my output nodes to map to anything; my concern is matching any possible output range. Its 0-10000 for my current problem typically, but i'm trying to find a general solution, of matching this to batch-normalised signals. $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 10, 2018 at 11:06
  • $\begingroup$ The wording of the question is still a bit confusing - are you asking how to bound the output/s of your neural network to any particular output range? $\endgroup$
    – rpatel
    Commented Jul 10, 2018 at 13:24
  • $\begingroup$ I am not sure what you mean by 'bound'. The outputs of my neural network should match the training data. Either I normalise my training data outside the network, or I have at least one last layer in the network that is not batch normalised and has non-saturating activation. I outline the drawbacks of both approaches in my original post; I am wondering if there are other options I am missing. $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 10, 2018 at 14:32

1 Answer 1

Reset to default
5
+25
$\begingroup$

You can't best practice your way out of a problem you didn't best practice your way into. Get rid of the multiplicative output node. Use a normal 1-node output layer with linear activation and do include a bias. This is the default recommendation for regression, for good reason.

Roughly speaking, for intuition purposes only, this is the same as doing a normal linear regression as the final step in your process. Linear regression always gives the best linear unbiased estimate. In particular, the mean of your predictions is pretty much guaranteed to match the mean of your training set.

All of the learning happens in the hidden layers. By the time we get to the output layer, we don't need to do any more work - there are already plenty of features on the second-to-last layer that have very high mutual information with the output. All we need to do in the output layer is a little book-keeping - averaging together features and centering and scaling them so match the response variable $y$. And the usual 1-node linear response using bias output layer is perfect for that.

Improve this answer
$\endgroup$
8
  • $\begingroup$ What you are saying makes a lot of sense; and this is what I started out trying. The problem is that it does not give me the expected results in practice. That is, it takes an exorbitant number of iterations for this linear scaling layer to converge if my output range differs from unit scale by a few orders of magnitude; that is the training of this uninitialized last layer dominates most of my training time requirements, which cant be right. cont. $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 11, 2018 at 14:37
  • $\begingroup$ But maybe my problem can be viewed as a learning-rate schedule problem? Indeed starting training with a very aggressive learning rate does seem to solve most of it; this lr is wildly different from the optimal one after the last layer is converged though; so this large lr to get the last layer right effectively scrambles the init of my hidden layers, and it does seem to influence my regression quality in subtle ways. It seems to be crucial here that my hidden layers are maxout so this wild-init-ride cant kill off lots of my gradients; switching to any saturating activation fn is indeed a nogo. $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 11, 2018 at 14:51
  • $\begingroup$ Another detail; my last layer cannot always contain a bias; the reason I am going with multiplication only is that I am usually striving to preserve positivity as a hard constraint. But using a bias or not does not seem to make a difference for my problems. Both best quality and best performance are still obtained by treating the optimisation of the denormalisation weights as a special-case. Am I missing something? Or should I change my question from finding to redefining best practices? $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 11, 2018 at 15:00
  • $\begingroup$ It sounds to me like it might be your loss function that's the problem, not the output node. Are you using MSE? $\endgroup$
    – olooney
    Commented Jul 11, 2018 at 15:09
  • $\begingroup$ I am using MAE; the greater sensitivity to outliers of MSE is very unwelcome in my application. $\endgroup$
    – Eelco Hoogendoorn
    Commented Jul 11, 2018 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.