Return to Answer

Of course one can consider a tensor product $V\otimes W$ of an $p$-dimensional $V$ and $q$-dimensional $W$ and choose to call its elements "tensors", but I am not aware of any useful such "tensor" and Wikipedia says that

But can we at least think of $X$ as a "generalized" tensor in the sense of an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). Linear transformations in $V$ correspond to linear transformation of features, and this is done in statistics all the time (think of PCA). But linear transformations in $W$ do not seem to correspond to anything meaningful: what would be a linear combination of people mean? So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

My conclusion is: machine learning "tensors" are not actually tensors in any meaningful or useful way.

Instead, they are multidimensional generalizations of matrices. Unfortunately, there is no established mathematical term for that.

Of course one can construct a tensor product $V\otimes W$ of an $p$-dimensional $V$ and $q$-dimensional $W$ but its elements are usually not called "tensors", as stated e.g. here on Wikipedia:

But can we at least think of $X$ as an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). A change of coordinates in $V$ corresponds to linear transformation of features, and this is done in statistics all the time (think of PCA). But a change of coordinates in $W$ does not seem to correspond to anything meaningful (and I urge anybody who has a counter-example to let me know in the comments). So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

And indeed, the common notation is to write $X\in\mathbb R^{n\times p}$, where $R^{n\times p}$ is a set of all $n\times p$ matrices (which, by the way, are defined as rectangular arrays of numbers, without any assumed transformation properties).

My conclusion is: (a) machine learning tensors are not math/physics tensors, and (b) it is mostly not useful to see them as elements of tensor products either.

Instead, they are multidimensional generalizations of matrices. Unfortunately, there is no established mathematical term for that, so it seems that this new meaning of "tensor" is now here to stay.

So in machine learning a tensor appears to be simply defined as a multidimensional numerical array. An example of such a 3D tensor would be $1000$ video frames of $640\times 480$ size. A usual $n\times p$ data matrix is an example of a 2D tensor according to this definition.

So in machine learning / data processing a tensor appears to be simply defined as a multidimensional numerical array. An example of such a 3D tensor would be $1000$ video frames of $640\times 480$ size. A usual $n\times p$ data matrix is an example of a 2D tensor according to this definition.

All tensors well-known in physics are like that: inertia tensor is $3\times 3$, electromagnetic tensor is $4\times 4$, curvature tensor is $4\times 4\times 4\times 4$.

Of course one can consider a tensor product $V\otimes W$ of an $n$-dimensional $V$ and $m$-dimensional $W$ and choose to call its elements "tensors", but I am not aware of any useful such "tensor" and Wikipedia says that

One example of a real tensor in statistics would be a covariance matrix. It is $p\times p$ and transforms in a particular way when the coordinate system in the $p$-dimensional feature space is changed. It is a tensor. But a $n\times p$ data matrix $X$ is not.

But can't we still think of $X$ as a tensor at least in a sense of an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). Linear transformations in $V$ correspond to linear transformation of features, this is done in statistics all the time (think of PCA). But linear transformations in $W$ do not seem to correspond to anything meaningful: what would be a linear combination of people mean? So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

Therefore my conclusion is: machine learning "tensors" are not actually tensors in any meaningful or useful way.

Instead, they are multidimensional generalizations of matrices.

All tensors well-known in physics are like that: inertia tensor in mechanics is $3\times 3$, electromagnetic tensor in special relativity is $4\times 4$, Riemann curvature tensor in general relativity is $4\times 4\times 4\times 4$. Curvature and electromagnetic tensors are actually tensor fields, which are sections of tensor bundles (see e.g. here but it gets technical), but all of that is defined over a vector space $V$.

Of course one can consider a tensor product $V\otimes W$ of an $p$-dimensional $V$ and $q$-dimensional $W$ and choose to call its elements "tensors", but I am not aware of any useful such "tensor" and Wikipedia says that

One example of a real tensor in statistics would be a covariance matrix. It is $p\times p$ and transforms in a particular way when the coordinate system in the $p$-dimensional feature space $V$ is changed. It is a tensor. But a $n\times p$ data matrix $X$ is not.

But can we at least think of $X$ as a "generalized" tensor in the sense of an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). Linear transformations in $V$ correspond to linear transformation of features, and this is done in statistics all the time (think of PCA). But linear transformations in $W$ do not seem to correspond to anything meaningful: what would be a linear combination of people mean? So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

My conclusion is: machine learning "tensors" are not actually tensors in any meaningful or useful way.

Instead, they are multidimensional generalizations of matrices. Unfortunately, there is no established mathematical term for that.