0
$\begingroup$

I have read a lot of articles to understand what is happening behind the scene in backpropagation like Ive gone through this and many other like that.

I think I understand how the backpropagation works, but not clear on some exceptions!.

now I want to explain the exception :)

my question is based on the proposed method in this paperhere which is a very interesting paper.

proposed method: big picture: they want to change the autoencoder in a way it gives a better representation of the data in the middle layer.

their idea is : KATE uses tanh activation function for the k-competitive hidden layer.

We divide these neurons into positive and negative neurons based on their activations. The most competitive k neurons are those that have the largest absolute activation values. However, we select the ⌈k/2⌉ largest positive activations as the positive winners, and reallocate the energy of the remaining positive loser neurons among the winners using an α amplification connection, where α is a hyperparameter. Finally, we set the activations of all losers to zero.

Similarly, the ⌊k/2⌋ lowest negative activations are the negative winners, and they incorporate the amplified energy from the negative loser neurons. We argue that the α amplification connections are a critical component in the kcompetitive layer. When α = 0, no gradients will flow through loser neurons, resulting in a regular k-sparse autoencoder (regardless of the activation functions and k-selection scheme). When α > 2/k, we actually boost the gradient signal flowing through the loser neurons. We empirically show that amplification helps improve the autoencoder model.

As an example, consider Figure 1, which shows an example feedforward step for k = 2. Here, h1 and h6 are the positive and negative winners, respectively, since the absolute activation potential for h1 is |z1 | = 0.8, and for h6 it s |z6 | = 0.6. The positive winner h1 takes away the energy from the positive losers h2 and h3, which is E({h2,h3}) = 0.2 + 0.1 = 0.3 Likewise, the negative winner h6 takes away the energy from the negative losers h4 and h5, which is −E({h4,h5}) = −(| − 0.1| + | −0.3|) = −0.4. The hyperparameter α governs how the energy from the loser neurons is incorporated into the winner neurons, for both positive and negative cases. That is h1’s net activation becomesz1 = 0.8 + 0.3α, and h6’s net activation is z6 = −0.6 − 0.4α. The rest of the neurons are set to zero activation.

Note that in the back-propagation procedure, the gradients will first flow through the winner neurons in the hidden layer and then the loser neurons via the α amplification connections. No gradients will flow directly from the output neurons to the loser neurons since they are made inactive in the feedforward step enter image description here

according to the bold part, how the neural net can firstly goes throgh the loser neuron then the winner!!! and also the loser neuron being set to zero how they can get some update regarding alpha??

Am I missing some part and not understanding backpropagation correctly?

Please let me know if the explanations are not clear. the link I shared is for the paper you just need to have a quick look on the section3

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You have to look at this layer as a 3 steps operation.

First $h_i = \tanh(W_i I)$ each hidden unit is a linear combination of the inputs with tanh activation. Then $pos = h_2 + h_3 = 0.3$ and $neg = h_4 + h_5 = - 0.4$ are computed. Finally, you build a modified hidden layer with units $\tilde{h_i}$: $\tilde{h_0} = h_0 + pos$, $\tilde{h_1} = 0$, ... $\tilde{h_6} = h_6 + neg$. And that modified layer is used as output of the k-competitive layer.

No gradient go through $\tilde{h_1}$ because it is set to $0$ as you said. However there will be gradient through $h_1$ via $\tilde{h_0}$ and $pos$. The gradient of the 'tilda' layer is first computed then backprpagated to $pos$ and $neg$ then backpropagated to the initial hidden layer $h_0 ... h_5$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thank you so much for your answer. I thought if a neuron is zero the gradient of other neurons will not flow to them, this is one point in which you made clear that will flow from other gradients. I greatly appreciate if you can provide your idea about the bold part in my question. how it can possible that gradient will firstly flow through winner neurons then the remining. again thank you so much for your explanations :) $\endgroup$
    – sariii
    Commented Oct 23, 2018 at 19:14
  • $\begingroup$ Actually, I can see that you made a point about my confusion part(bold part in the question) in the last line. but simply I can not understand that or better say how? I appreciate it if you can explain it more $\endgroup$
    – sariii
    Commented Oct 23, 2018 at 19:16
  • 1
    $\begingroup$ You can see it as three consecutive layers, - The first is $h_0 .. h_6$ with connections from the input layers. - The second is pos and neg with connections from the first layer - The third is the 'tilda layers' with connections from the first and second layers This third layer is the output of the "k-competitive layer" $\endgroup$
    – François Darmon
    Commented Oct 24, 2018 at 8:45
  • $\begingroup$ Thank you so much @user7573566 . I did not get this way by reading the paper. many thanks for your help $\endgroup$
    – sariii
    Commented Oct 24, 2018 at 16:08
  • 1
    $\begingroup$ I said "you can see it as", i do not know if they implemented three separate layers but it is useful to represent if with three layers. In fact any time you perform an operation, you can see it as a new layer. $\endgroup$
    – François Darmon
    Commented Oct 25, 2018 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.