# back propagation in neurons with zero weight and some specific conditions

I have read a lot of articles to understand what is happening behind the scene in backpropagation like Ive gone through this and many other like that.

I think I understand how the backpropagation works, but not clear on some exceptions!.

now I want to explain the exception :)

my question is based on the proposed method in this paperhere which is a very interesting paper.

proposed method: big picture: they want to change the autoencoder in a way it gives a better representation of the data in the middle layer.

their idea is : KATE uses tanh activation function for the k-competitive hidden layer.

We divide these neurons into positive and negative neurons based on their activations. The most competitive k neurons are those that have the largest absolute activation values. However, we select the ⌈k/2⌉ largest positive activations as the positive winners, and reallocate the energy of the remaining positive loser neurons among the winners using an α amplification connection, where α is a hyperparameter. Finally, we set the activations of all losers to zero.

Similarly, the ⌊k/2⌋ lowest negative activations are the negative winners, and they incorporate the amplified energy from the negative loser neurons. We argue that the α amplification connections are a critical component in the kcompetitive layer. When α = 0, no gradients will flow through loser neurons, resulting in a regular k-sparse autoencoder (regardless of the activation functions and k-selection scheme). When α > 2/k, we actually boost the gradient signal flowing through the loser neurons. We empirically show that amplification helps improve the autoencoder model.

As an example, consider Figure 1, which shows an example feedforward step for k = 2. Here, h1 and h6 are the positive and negative winners, respectively, since the absolute activation potential for h1 is |z1 | = 0.8, and for h6 it s |z6 | = 0.6. The positive winner h1 takes away the energy from the positive losers h2 and h3, which is E({h2,h3}) = 0.2 + 0.1 = 0.3 Likewise, the negative winner h6 takes away the energy from the negative losers h4 and h5, which is −E({h4,h5}) = −(| − 0.1| + | −0.3|) = −0.4. The hyperparameter α governs how the energy from the loser neurons is incorporated into the winner neurons, for both positive and negative cases. That is h1’s net activation becomesz1 = 0.8 + 0.3α, and h6’s net activation is z6 = −0.6 − 0.4α. The rest of the neurons are set to zero activation.

Note that in the back-propagation procedure, the gradients will first flow through the winner neurons in the hidden layer and then the loser neurons via the α amplification connections. No gradients will flow directly from the output neurons to the loser neurons since they are made inactive in the feedforward step

according to the bold part, how the neural net can firstly goes throgh the loser neuron then the winner!!! and also the loser neuron being set to zero how they can get some update regarding alpha??

Am I missing some part and not understanding backpropagation correctly?

Please let me know if the explanations are not clear. the link I shared is for the paper you just need to have a quick look on the section3

First $$h_i = \tanh(W_i I)$$ each hidden unit is a linear combination of the inputs with tanh activation. Then $$pos = h_2 + h_3 = 0.3$$ and $$neg = h_4 + h_5 = - 0.4$$ are computed. Finally, you build a modified hidden layer with units $$\tilde{h_i}$$: $$\tilde{h_0} = h_0 + pos$$, $$\tilde{h_1} = 0$$, ... $$\tilde{h_6} = h_6 + neg$$. And that modified layer is used as output of the k-competitive layer.
No gradient go through $$\tilde{h_1}$$ because it is set to $$0$$ as you said. However there will be gradient through $$h_1$$ via $$\tilde{h_0}$$ and $$pos$$. The gradient of the 'tilda' layer is first computed then backprpagated to $$pos$$ and $$neg$$ then backpropagated to the initial hidden layer $$h_0 ... h_5$$.
• You can see it as three consecutive layers, - The first is $h_0 .. h_6$ with connections from the input layers. - The second is pos and neg with connections from the first layer - The third is the 'tilda layers' with connections from the first and second layers This third layer is the output of the "k-competitive layer" Oct 24, 2018 at 8:45