Currently the transformer blocks are of this structure (ignore all norm's, masks, dropout ...):
x
y1 = x + attention(x)
y2 = y1 + linear(y1)
Why not adding to x one change, not two different ?
x
y1 = x + attention(x)
y2 = x + linear(y1)
not seen this alternative.
Is it obvious that is bad ?
The transformer's consistent use of residual connections helps maintain the flow of information and gradients throughout the network which is crucial for training deep models. By always adding the output of a sub-layer back to its input the network can learn to refine its representations layer by layer without losing the original input information.
Your proposed structure modifies this pattern by adding the result of the feed-forward network directly to the original input rather than the output of the attention layer. This breaks the consistency seen in traditional transformers because it bypasses the residual connection's typical role of accumulating modifications in a sequential manner. Instead it resets the baseline to the original input before applying the feed-forward output which disrupts gradients flow through the network during backpropagation possibly making optimization more difficult or unstable.
Of course this isn't necessarily to prevent you from exploring this alternative structure empirically with some experiments to see how it impacts training stability and model performance. This might require different tuning to be effective and could potentially learn different kinds of representations if successful.
