# Multiplication, addition, and concatenation in deep neural networks

In conveying information between layers/nodes/neurons in a deep neural network one can choose between multiplication, addition, and concatenation.

So, lets say that we have an input which passes the data to two, different, layers ($L_1$ and $L_2$) and these layers have as output a vector of size $1xM_1$ for $L_1$ and $1xM_2$ for $L_2$.

Then, we have another layer, $L_3$, to which we want to pass the information of the $L_1$ and $L_2$. What would be the difference of using addition or concatenation? I know that multiplication is used to weight the information to be conveyed. But what about addition and concatenation? What is the conceptual/model-wise result in the information conveyance?

E.g., in https://arxiv.org/abs/1606.03475, figure 1, we used concatenation to create the token emdeddings $e_i$ from the characters as we want to motivate the higher layers to consider the information from both the forward character-based RNN and the backward character-based RNN.