5
$\begingroup$

In conveying information between layers/nodes/neurons in a deep neural network one can choose between multiplication, addition, and concatenation.

So, lets say that we have an input which passes the data to two, different, layers ($L_1$ and $L_2$) and these layers have as output a vector of size $1xM_1$ for $L_1$ and $1xM_2$ for $L_2$.

Then, we have another layer, $L_3$, to which we want to pass the information of the $L_1$ and $L_2$. What would be the difference of using addition or concatenation? I know that multiplication is used to weight the information to be conveyed. But what about addition and concatenation? What is the conceptual/model-wise result in the information conveyance?

$\endgroup$
3
+50
$\begingroup$

Addition and concatenation are special cases of multiplication, where the weights are equal to 0 or 1. As a result, one can view using addition and concatenation as assumptions of what the network should be doing.

E.g., in https://arxiv.org/abs/1606.03475, figure 1, we used concatenation to create the token emdeddings $e_i$ from the characters as we want to motivate the higher layers to consider the information from both the forward character-based RNN and the backward character-based RNN.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.