Facebook's infersent intuition

Question

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with:

1. (u, v) the concatenation between u and v,
2. u * v the element-wise product,
3. |u - v| the absolute element-wise difference

While I can somewhat get a feel for why points 1 and 3 were used, I do not really understand how an element-wise product can help in this case.

My intuition for 1 and 3 is the following:

• Point 1 was used because it makes sense to feed the actual encodings of the two sentences
• The element-wise difference gives a sense of the similarity between the two sentence encodings

Does anyone know, why the element-wise product would help?

PS: A picture of the infersent architecture can be found below (extracted from the paper).

Answer 1

First of all, many tricks in deep learning are used because they were "proved to work", with post factum theoretical rationalizations. So in many cases the "why" questions can be only answered in terms of "why do we think that this worked", rather then any deeper rationale behind it.

In this case, the network encodes both sentences using same encoder, obtaining two vector representations $u$, $v$ for two sentences. They are feed into fully-connected network as features:

• with concatenation $(u, v)$, they take both vectors as-is, so the representation of each sentence is a feature,
• with absolute difference $|u-v|$, they look at magnitude of differences between "features" (elements) of each vector, e.g. say that $i$-th element recognizes that sentence is a question, so $|u_i - v_i|$ is how much both sentences differ in terms of being a question or not,
• element-wise product $u*v$, is basically an interaction term, this can catch similarities between values (big * big = bigger; small * small = smaller), or the discrepancies (negative * positive = negative) (see example here).

This is a feature engineering build-in the architecture of the network. It is a kind of thing that may not exactly be needed and could be figured out on it's own by a big enough network, but may speed up and simplify the computation. Same with choice of the aggregating functions, in other cases, different combination of them may work better, so you could experiment with them as well.