Is it possible to have negative weights (after enough epochs) for deep convolutional neural networks when we use ReLU for all the activation layers?

  • $\begingroup$ I don't see any reason why they can't be negative. Is there a particular reason/observation that you have in mind? $\endgroup$
    – Sobi
    Dec 8 '15 at 23:27
  • $\begingroup$ I am just imagining the process of SGD and think about whether negative weight is common and possible. $\endgroup$ Dec 9 '15 at 22:01
  • $\begingroup$ He thought to that because "weight" resemble synapses, the links between the neurons, so how can we have -2 synapses towards a neuron?? I stumbled here after searching the exact same thing on google... I guess it could be possible anyway, it could mean eventually a missing synapse or link and the "hops" to reach the b from another side that go subtracted from the calculus, but I'm not really sure, just thinking $\endgroup$
    – Try
    Feb 4 '19 at 21:21

Rectified Linear Units (ReLUs) only make the output of the neurons to be non-negative. The parameters of the network, however, can, and will, become positive or negative depending on the training data.

Here are two reasons I can think of right now that justifies (intuitively) why some parameters would become negative:

  1. the regularization of the parameters (a.k.a. the weight decay); the variation in the parameter values makes prediction possible, and if the parameters are centered around zero (i.e. their mean is close to zero), then their $\ell 2$ norm (which is a standard regularizer) is low.

  2. although the gradients of the output of a layer with respect to the layer parameters depend on the input to the layer (which are always positive assuming that the previous layer passes its outputs through a ReLU), however, the gradient of the error (which comes from the layers closer to the final output layers) may be positive or negative, making it possible for SGD to make some of the parameter values negative after taking the next gradient step. More specifically, let $I$, $O$, and $w$ denote the input, output, and parameters of a layer in a neural network. Also, let $E$ be the final error of the network induced by some training sample. The gradient of the error with respect to $w$ is computed as $\frac{\partial E}{\partial w} = \left( \sum_{k=1}^K\frac{\partial E}{\partial O_k} \right) \cdot \frac{\partial O_k}{\partial w}$; note that $O_k = O, \forall k$ (see picture below):

enter image description here


Imagine that you have optimal weights which are all non-negative.

Now invert some input variable $x'_i = -x_i$. The optimal network for this setup is with the weights of the $\{x'_i,y\}$ edges inverted, so the new weights are non-positive.


Unless you use another activation function for instance Leaky ReLU. Rectified weights of layers after the first one are non-negative regardless how many epochs in training.

  • 1
    $\begingroup$ Great thanks! Can you explain a bit more detials on how Leaky ReLU can lead to negative weight? $\endgroup$ Dec 5 '15 at 20:23
  • $\begingroup$ Seems like the claim is not true. I have run training on ReLU activated network, the matrices in the affine transformation ("Ws") and the offsets ("b's"), which I presume refereed in this question as the weights, do get negative values. $\endgroup$
    – them
    Jan 6 '17 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.