I wasn't entire sure if this question belongs here or on math.stackexchange, since it's only partially about machine learning, but perhaps more of a conceptual question about linear algebra (or tensor algebra). Please let me know if you believe this question is better asked on another SE site.
The type of network I am reading about is described for example here:
Socher et al: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
By my current level of understanding, the basic network architecture is very similar to a recursive neural network, i.e. a recurrent neural network where the unfolded network given some finite input is expressed as a (usually: binary) tree, instead of a "flat" chain (as in the recurrent network).
The difference then rests in the composition function at each of the tree's nodes. The parent vector $p$ of two children vectors (either input words, or parent vectors themselves), is calculated as follows:
$p = f(\begin{bmatrix} b \\ c \end{bmatrix}^T V^{[1:d]} \begin{bmatrix} b \\ c \end{bmatrix} + W\begin{bmatrix} b \\ c \end{bmatrix})$
Here, $f$ is a nonlinearity like tanh, $b$, $c$ are $d$-dimensional children vectors, and $V^{[1:d]}$ is one of $d$ 'slices' of the tensor (author's terminology), and the $W\begin{bmatrix} b \\ c \end{bmatrix}$ is the usual (non-tensor network) standard layer multiplication added at the end.
Please correct me if I made any mistakes so far.
My question is about what exactly changes, in terms of "expressive power" and the network's ability to learn, when we go from a single weight matrix to "slices of a tensor" as above.
I'll try to break down my uncertainty about this approach into smaller partial questions:
In principle, the units (of a standard multi-layer ANN, just as well as those of a "tree-shaped" RNN) are already able to capture any function we are interested in. Given a sufficient number of hidden units, a network approximates a non-linear function as a linear combination of the non-linearity (like tanh) - if this function is a universal basis function. Under this view, I am wondering what the additional "power" of the tensor-based units is?
One difference that I can see is that the $d$ "slices" of a tensor are essentially a way to increase the dimensionality of the units and the network in total, while leaving the input dimensionality unchanged. Instead of a $[d \, \times \, 2d]$ matrix, our units now include $d$ times a $[2d \, \times \, 2d]$ matrix ($V$ above). This to me looks like the network simply has more dimensions for learning the same input - but as a trade-off, this could also increase the risk of overfitting. Correct?
In a lecture script, Socher remarks on the difference between the two that in a regular recursive network, the interaction of the two input vectors is "implicit" only (through the non-linearity), while in tensor networks, the interaction between them is "multiplicative". I am not sure what he means by that. In the regular recursive network, two $d$-dimensional input vectors are concatenated, then multiplied by a $[d \, \times \, 2d]$ matrix, resulting in a $d$ dimensional vector (that can be used as input for the next tree node again). In the tensor network above, the two vectors are individually multiplied by a $[2d \, \times \, 2d]$ matrix, resulting in a scalar - and we then "reconstruct" the original vector size by using $d$ such tensor "slices". I seem to miss how in the first case the vectors "interact" less than in the second case - could anyone give me the intuition here?
Finally, a natural way of thinking of the learning power of networks seems to be in terms of graphical "projections", i.e. linear transformations like e.g. rotation or reflection. My formal understanding of tensors is limited, especially in the case of the tensor "slices" approach taken by Socher. As a result, I don't know whether this tensor approach allows for more "complex" transformations than those possible in a regular recursive network (using only a single matrix). More specifically, it seems that because of the additional "slices", we have more dimensions to work with (as mentioned above), but that the transformations are still essentially the same we had in the single matrix case, i.e. linear transformations. Is this view correct?
Thank you for reading this (rather long) question. Any remarks or corrections would be appreciated.
