I am using "add" and "concatenate" as it is defined in keras. Basically, from my understanding,
add will sum the inputs (which are the layers, in essence tensors). So if the first layer had a particular weight as
0.4 and another layer with the same exact shape had the corresponding weight being
0.5, then after the
add the new weight becomes
However, with concatenate, let's say the first layer has dimensions
64x128x128 and the second layer had dimensions
32x128x128, then after concatenate, the new dimensions are
96x128128 (assuming you pass in the second layer as the first input into concatenate).
Assuming my above intuition is true, when would I use one over the other? Conceptually,
add seems a sharing of information that potentially results in information distortion while concatenate is a sharing of information in the literal sense.