I have been reading a Facebook paper, read here, and am confused about certain features of the architecture. I do not understand why they have a tanh layer, max-pooling layer, and then another tanh layer. I understand what each layer does, but I don't understand why they have this sequence. Wouldn't this output basically give the same as just a max-pooling layer, and a tanh layer?

#Tagspace Architecture


Tanh layers include their own weight matrices. Max pooling selects elements from different "channels." So the order of operations in the tanh-max-tanh pipeline is

  1. linear transformation
  2. $\text{tanh}$
  3. select elements according to max pooling, discarding the rest
  4. linear transformation
  5. $\text{tanh}$

That is, the "down-selecting" to the max pooling elements means that you're looking at a different set of vectors under linear combination than you would in the alternative max pool-tanh case.

Finally, remember that the linear combinations can reverse the sign of the inputs! This is important because if you flip the sign of inputs to the max pooling, you'll get different results.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.