I use two LSTM encoders to encode two inputs from two different sources(with different time steps or length) and I get a pair of outputs and a pair of output states from each encoder. I concatenate the two outputs for the attention mechanism of the decoder, but I don't know how to combine the two output states to input it to the decoder. Should I concatenate them(to scale up the dimension to be twice as before) or keep the dimension by doing the simple element-wise addition? Or any other methods? But what is the mathematics underpinning the choice? Or is it totally empirical?
I cannot speak to the mathematics but most logical path is to concatenate two outputs together effectively doubling the dimensions. Another way is to insert dense layers after each LSTM output and combine them later. This way you can find number of dimensions that works the best overall and will potentially be able to tweak dropouts on each section of your net