When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with:

1. (u, v) the concatenation between u and v,
2. u * v the element-wise product,
3. |u - v| the absolute element-wise difference

While I can somewhat get a feel for why points 1 and 3 were used, I do not really understand how an element-wise product can help in this case.

My intuition for 1 and 3 is the following:

• Point 1 was used because it makes sense to feed the actual encodings of the two sentences
• The element-wise difference gives a sense of the similarity between the two sentence encodings

Does anyone know, why the element-wise product would help?

PS: A picture of the infersent architecture can be found below (extracted from the paper).

• Could you make it more clear what exactly are you referring to? Maybe you could post the exact formulas? – Tim Jan 7 '19 at 8:47
• @Tim I added a graph showing the actual architecture – ryuzakinho Jan 7 '19 at 8:59

In this case, the network encodes both sentences using same encoder, obtaining two vector representations $$u$$, $$v$$ for two sentences. They are feed into fully-connected network as features:
• with concatenation $$(u, v)$$, they take both vectors as-is, so the representation of each sentence is a feature,
• with absolute difference $$|u-v|$$, they look at magnitude of differences between "features" (elements) of each vector, e.g. say that $$i$$-th element recognizes that sentence is a question, so $$|u_i - v_i|$$ is how much both sentences differ in terms of being a question or not,
• element-wise product $$u*v$$, is basically an interaction term, this can catch similarities between values (big * big = bigger; small * small = smaller), or the discrepancies (negative * positive = negative) (see example here).