# When to “add” layers and when to “concatenate” in neural networks?

I am using "add" and "concatenate" as it is defined in keras. Basically, from my understanding, add will sum the inputs (which are the layers, in essence tensors). So if the first layer had a particular weight as 0.4 and another layer with the same exact shape had the corresponding weight being 0.5, then after the add the new weight becomes 0.9.

However, with concatenate, let's say the first layer has dimensions 64x128x128 and the second layer had dimensions 32x128x128, then after concatenate, the new dimensions are 96x128128 (assuming you pass in the second layer as the first input into concatenate).

Assuming my above intuition is true, when would I use one over the other? Conceptually, add seems a sharing of information that potentially results in information distortion while concatenate is a sharing of information in the literal sense.

Note that $W[x,y] = W_1x + W_2y$ where $[\ ]$ denotes concat and $W$ is split horizontally into $W_1$ and $W_2$. Compare this to $W(x+y) = Wx + Wy$. So you can interpret adding as a form of concatenation where the two halves of the weight matrix are constrained to $W_1 = W_2$.