This is not an answer to your question, but an extended comment on the issue that has been raised here in comments by different people, namely: **are machine learning "tensors" the same thing as tensors in mathematics?**

Now, according to the Cichoki 2014, [Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions](http://arxiv.org/pdf/1403.2048.pdf), and Cichoki et al. 2014, [Tensor Decompositions for Signal Processing Applications](http://arxiv.org/pdf/1403.4462.pdf),

> A higher-order tensor can be interpreted as a multiway
array, [...]

> A tensor can be thought of as a multi-index numerical array, [...]

> Tensors (i.e., multi-way arrays) [...]

So in machine learning *a tensor* appears to be simply defined as a multidimensional numerical array. An example of such a 3D tensor would be $1000$ video frames of $640\times 480$ size. A usual $n\times p$ data matrix is an example of a 2D tensor according to this definition.

**This is not how tensors are defined in mathematics and physics!**

A tensor can be defined as a multidimensional array obeying certain transformation laws under the change of coordinates ([see Wikipedia](https://en.wikipedia.org/wiki/Tensor#As_multidimensional_arrays) or the first sentence in [MathWorld article](http://mathworld.wolfram.com/Tensor.html)). A better but equivalent definition ([see Wikipedia](https://en.wikipedia.org/wiki/Tensor#Using_tensor_products)) says that a tensor on vector space $V$ is an element of $V\otimes\ldots\otimes V^*$. Note that this means that, when represented as multidimensional arrays, tensors are of size $p\times p$ or $p\times p\times p$ etc., where $p$ is the dimensionality of $V$.

All tensors well-known in physics are like that: [inertia tensor](https://en.wikipedia.org/wiki/Moment_of_inertia#The_inertia_tensor) is $3\times 3$, [electromagnetic tensor](https://en.wikipedia.org/wiki/Electromagnetic_tensor) is $4\times 4$, [curvature tensor](https://en.wikipedia.org/wiki/Riemann_curvature_tensor) is $4\times 4\times 4\times 4$.

Of course one can consider a tensor product $V\otimes W$ of an $n$-dimensional $V$ and $m$-dimensional $W$ and choose to call its elements "tensors", but I am not aware of *any* useful such "tensor" and Wikipedia [says that](https://en.wikipedia.org/wiki/Tensor#Using_tensor_products)

> In principle, one could define a "tensor" simply to be an element of any tensor product. However, the mathematics literature usually reserves the term tensor for an element of a tensor product of a single vector space $V$ and its dual, as above.

One example of a real tensor in statistics would be a covariance matrix. It is $p\times p$ and transforms in a particular way when the coordinate system in the $p$-dimensional feature space is changed. It is a tensor. But a $n\times p$ data matrix $X$ is not.

But can't we still think of $X$ as a tensor at least in a sense of an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). Linear transformations in $V$ correspond to linear transformation of features, this is done in statistics all the time (think of PCA). But linear transformations in $W$ do not seem to correspond to anything meaningful: what would be a linear combination *of people* mean? So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

**Therefore my conclusion is: machine learning "tensors" are not actually tensors in any meaningful or useful way.**

Instead, they are multidimensional generalizations of matrices.