I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world suddenly fascinated with tensors? Are there recent papers/ standard results that are particularly surprising, that brought about this? Is it computationally a lot cheaper than previously suspected?

I'm not being glib, I sincerely am interested, and if there are any pointers to papers about this, I'd love to read them.

  • 33
    $\begingroup$ It seems like the only retaining feature that "big data tensors" share with the usual mathematical definition is that they are multidimensional arrays. So I'd say that big data tensors are a marketable way of saying "multidimensional array," because I highly doubt that machine learning people will care about either the symmetries or transformation laws that the usual tensors of mathematics and physics enjoy, especially their usefulness in forming coordinate free equations. $\endgroup$
    – Alex R.
    Commented Feb 23, 2016 at 19:00
  • 2
    $\begingroup$ @AlexR. without invariance to transformations there are no tensors $\endgroup$
    – Aksakal
    Commented Feb 23, 2016 at 21:43
  • 3
    $\begingroup$ Putting on my mathematical hat I can say that there is no intrinsic symmetry to a mathematical tensor. Further, they are another way to say 'multidimensional array'. One could vote for using the word tensor over using the phrase multidimensional array simply on grounds of simplicity. In particular if V is a n - dimensional vector space, one can identify $V \otimes V$ with n by n matrices. $\endgroup$
    – meh
    Commented Feb 24, 2016 at 14:47
  • 3
    $\begingroup$ @aginensky If a tensor were nothing more than a multidimensional array, then why do the definitions of tensors found in math textbooks sound so complicated? From Wikipedia: "The numbers in the multidimensional array are known as the scalar components of the tensor... Just as the components of a vector change when we change the basis of the vector space, the components of a tensor also change under such a transformation. Each tensor comes equipped with a transformation law that details how the components of the tensor respond to a change of basis." In math, a tensor is not just an array. $\endgroup$
    – littleO
    Commented Feb 25, 2016 at 0:55
  • 5
    $\begingroup$ Just some general thoughts on this discussion: I think that, as with vectors and matrices, the actual application often becomes a much-simplified instantiation of much richer theory. I am reading this paper in more depth: epubs.siam.org/doi/abs/10.1137/07070111X?journalCode=siread and one thing that is really impressing me is that the "representational" tools for matrices (eigenvalue and singular value decompositions) have interesting generalizations in higher orders. I'm sure there are many more beautiful properties as well, beyond just a nice container for more indices. :) $\endgroup$
    – Y. S.
    Commented Feb 25, 2016 at 15:40

10 Answers 10


This is not an answer to your question, but an extended comment on the issue that has been raised here in comments by different people, namely: are machine learning "tensors" the same thing as tensors in mathematics?

Now, according to the Cichoki 2014, Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions, and Cichoki et al. 2014, Tensor Decompositions for Signal Processing Applications,

A higher-order tensor can be interpreted as a multiway array, [...]

A tensor can be thought of as a multi-index numerical array, [...]

Tensors (i.e., multi-way arrays) [...]

So called tensors in machine learning

So in machine learning / data processing a tensor appears to be simply defined as a multidimensional numerical array. An example of such a 3D tensor would be $1000$ video frames of $640\times 480$ size. A usual $n\times p$ data matrix is an example of a 2D tensor according to this definition.

This is not how tensors are defined in mathematics and physics!

A tensor can be defined as a multidimensional array obeying certain transformation laws under the change of coordinates (see Wikipedia or the first sentence in MathWorld article). A better but equivalent definition (see Wikipedia) says that a tensor on vector space $V$ is an element of $V\otimes\ldots\otimes V^*$. Note that this means that, when represented as multidimensional arrays, tensors are of size $p\times p$ or $p\times p\times p$ etc., where $p$ is the dimensionality of $V$.

All tensors well-known in physics are like that: inertia tensor in mechanics is $3\times 3$, electromagnetic tensor in special relativity is $4\times 4$, Riemann curvature tensor in general relativity is $4\times 4\times 4\times 4$. Curvature and electromagnetic tensors are actually tensor fields, which are sections of tensor bundles (see e.g. here but it gets technical), but all of that is defined over a vector space $V$.

Of course one can construct a tensor product $V\otimes W$ of an $p$-dimensional $V$ and $q$-dimensional $W$ but its elements are usually not called "tensors", as stated e.g. here on Wikipedia:

In principle, one could define a "tensor" simply to be an element of any tensor product. However, the mathematics literature usually reserves the term tensor for an element of a tensor product of a single vector space $V$ and its dual, as above.

One example of a real tensor in statistics would be a covariance matrix. It is $p\times p$ and transforms in a particular way when the coordinate system in the $p$-dimensional feature space $V$ is changed. It is a tensor. But a $n\times p$ data matrix $X$ is not.

But can we at least think of $X$ as an element of tensor product $W\otimes V$, where $W$ is $n$-dimensional and $V$ is $p$-dimensional? For concreteness, let rows in $X$ correspond to people (subjects) and columns to some measurements (features). A change of coordinates in $V$ corresponds to linear transformation of features, and this is done in statistics all the time (think of PCA). But a change of coordinates in $W$ does not seem to correspond to anything meaningful (and I urge anybody who has a counter-example to let me know in the comments). So it does not seem that there is anything gained by considering $X$ as an element of $W\otimes V$.

And indeed, the common notation is to write $X\in\mathbb R^{n\times p}$, where $R^{n\times p}$ is a set of all $n\times p$ matrices (which, by the way, are defined as rectangular arrays of numbers, without any assumed transformation properties).

My conclusion is: (a) machine learning tensors are not math/physics tensors, and (b) it is mostly not useful to see them as elements of tensor products either.

Instead, they are multidimensional generalizations of matrices. Unfortunately, there is no established mathematical term for that, so it seems that this new meaning of "tensor" is now here to stay.

  • 26
    $\begingroup$ I am a pure mathematician, and this is a very good answer. In particular, the example of a covariance matrix is an excellent way to understand the "transformation properties" or "symmetries" that seemed to cause confusion above. If you change coordinates on your $p$-dimensional feature space, the covariance matrix transforms in a particular and possibly surprising way; if you did the more naive transformation on your covariances you would end up with incorrect results. $\endgroup$
    – Tom Church
    Commented Feb 25, 2016 at 4:08
  • 12
    $\begingroup$ Thanks, @Tom, I appreciate that you registered on CrossValidated to leave this comment. It has been a long time since I was studying differential geometry so I am happy if somebody confirms what I wrote. It is a pity that there is no established term in mathematics for "multidimensional matrices"; it seems that "tensor" is going to stick in machine learning community as a term for that. How do you think one should rather call it though? The best thing that comes to my mind is $n$-matrices (e.g. $3$-matrix to refer to a video object), somewhat analogously to $n$-categories. $\endgroup$
    – amoeba
    Commented Feb 25, 2016 at 10:25
  • 5
    $\begingroup$ @amoeba, in programming the multidemensional matrices are usually called arrays, but some languages such as MATLAB would call them matrices. For instance, in FORTRAN the arrays can have more than 2 dimensions. In languages like C/C++/Java the arrays are one dimensional, but you can have arrays of arrays, making them work like multidimensional arrays too. MATLAB supports 3 or more dimensional arrays in the syntax. $\endgroup$
    – Aksakal
    Commented Feb 25, 2016 at 15:17
  • 3
    $\begingroup$ That is very interesting. I hope you will emphasize that point. But please take some care not to confuse a set with a vector space it determines, because the distinction is important in statistics. In particular (to pick up one of your examples), although a linear combination of people is meaningless, a linear combination of real-valued functions on a set of people is both meaningful and important. It's the key to solving linear regression, for instance. $\endgroup$
    – whuber
    Commented Jul 5, 2016 at 13:56
  • 8
    $\begingroup$ Per T. Kolda, B, Bada,"Tensor Decompositions and Applications" SIAM Review 2009, epubs.siam.org/doi/pdf/10.1137/07070111X 'A tensor is a multidimensional array. More formally, an N-way or Nth-order tensor is an element of the tensor product of N vector spaces, each of which has its own coordinate system. This notion of tensors is not to be confused with tensors in physics and engineering (such as stress tensors),, which are generally referred to as tensor fields in mathematics " $\endgroup$ Commented Sep 3, 2016 at 20:31

Tensors often offer more natural representations of data, e.g., consider video, which consists of obviously correlated images over time. You can turn this into a matrix, but it's just not natural or intuitive (what does a factorization of some matrix-representation of video mean?).

Tensors are trending for several reasons:

  • our understanding of multilinear algebra is improving rapidly, specifically in various types of factorizations, which in turn helps us to identify new potential applications (e.g., multiway component analysis)
  • software tools are emerging (e.g., Tensorlab) and are being welcomed
  • Big Data applications can often be solved using tensors, for example recommender systems, and Big Data itself is hot
  • increases in computational power, as some tensor operations can be hefty (this is also one of the major reasons why deep learning is so popular now)
  • 9
    $\begingroup$ On the computational power part: I think the most important is that linear algebra can be very fast on GPUs, and lately they have gotten bigger and faster memories, that is the biggest limitation when processing large data. $\endgroup$
    – Davidmh
    Commented Feb 23, 2016 at 12:17
  • 6
    $\begingroup$ Marc Claesen's answer is a good one. David Dunson, Distinguished Professor of Statistics at Duke, has been one of the key exponents of tensor-based approaches to modeling as in this presentation, Bayesian Tensor Regression. icerm.brown.edu/materials/Slides/sp-f12-w1/… $\endgroup$
    – user78229
    Commented Feb 23, 2016 at 14:36
  • 1
    $\begingroup$ As mentioned by David, Tensor algorithms often lend themselves well to parallelism, which hardware (such as GPU accelerators) are increasingly getting better at. $\endgroup$ Commented Feb 23, 2016 at 15:15
  • 1
    $\begingroup$ I assumed that the better memory/CPU capabilities were playing a part, but the very recent burst of attention was interesting; I think it must be because of a lot of recent surprising successes with recommender systems, and perhaps also kernels for SVMs, etc. Thanks for the links! great places to start learning about this stuff... $\endgroup$
    – Y. S.
    Commented Feb 24, 2016 at 7:20
  • 6
    $\begingroup$ If you store a video as a multidimensional array, I don't see how this multidimensional array would have any of the invariance properties a tensor is supposed to have. It doesn't seem like the word "tensor" is appropriate in this example. $\endgroup$
    – littleO
    Commented Feb 24, 2016 at 8:45

I think your question should be matched with an answer that is equally free flowing and open minded as the question itself. So, here they are my two analogies.

First, unless you're a pure mathematician, you were probably taught univariate probabilities and statistics first. For instance, most likely your first OLS example was probably on a model like this: $$y_i=a+bx_i+e_i$$ Most likely, you went through deriving the estimates through actually minimizing the sum of least squares: $$TSS=\sum_i(y_i-\bar a-\bar b x_i)^2$$ Then you write the FOCs for parameters and get the solution: $$\frac{\partial TTS}{\partial \bar a}=0$$

Then later you're told that there's an easier way of doing this with vector (matrix) notation: $$y=Xb+e$$

and the TTS becomes: $$TTS=(y-X\bar b)'(y-X\bar b)$$

The FOCs are: $$2X'(y-X\bar b)=0$$

And the solution is $$\bar b=(X'X)^{-1}X'y$$

If you're good at linear algebra, you'll stick to the second approach once you've learned it, because it's actually easier than writing down all the sums in the first approach, especially once you get into multivariate statistics.

Hence my analogy is that moving to tensors from matrices is similar to moving from vectors to matrices: if you know tensors some things will look easier this way.

Second, where do the tensors come from? I'm not sure about the whole history of this thing, but I learned them in theoretical mechanics. Certainly, we had a course on tensors, but I didn't understand what was the deal with all these fancy ways to swap indices in that math course. It all started to make sense in the context of studying tension forces.

So, in physics they also start with a simple example of pressure defined as force per unit area, hence: $$F=p\cdot dS$$ This means you can calculate the force vector $F$ by multiplying the pressure $p$ (scalar) by the unit of area $dS$ (normal vector). That is when we have only one infinite plane surface. In this case there's just one perpendicular force. A large balloon would be good example.

However, if you're studying tension inside materials, you are dealing with all possible directions and surfaces. In this case you have forces on any given surface pulling or pushing in all directions, not only perpendicular ones. Some surfaces are torn apart by tangential forces "sideways" etc. So, your equation becomes: $$F=P\cdot dS$$ The force is still a vector $F$ and the surface area is still represented by its normal vector $dS$, but $P$ is a tensor now, not a scalar.

Ok, a scalar and a vector are also tensors :)

Another place where tensors show up naturally is co-variance or correlation matrices. Just think of this: how to transform one correlation matrix $C_0$ to another one $C_1$? You realize we can't just do it this way: $$C_\theta(i,j)=C_0(i,j)+ \theta(C_1(i,j)-C_0(i,j)),$$ where $\theta\in[0,1]$ because we need to keep all $C_\theta$ positive semi-definite.

So, we'd have to find the path $\delta C_\theta$ such that $C_1=C_0+\int_\theta\delta C_\theta$, where $\delta C_\theta$ is a small disturbance to a matrix. There are many different paths, and we could search for the shortest ones. That's how we get into Riemannian geometry, manifolds, and... tensors.

UPDATE: what's tensor, anyway?

@amoeba and others got into a lively discussion of the meaning of tensor and whether it's the same as an array. So, I thought an example is in order.

Say, we go to a bazaar to buy groceries, and there are two merchant dudes, $d_1$ and $d_2$. We noticed that if we pay $x_1$ dollars to $d_1$ and $x_2$ dollars to $d_2$ then $d_1$ sells us $y_1=2x_1-x_2$ pounds of apples, and $d_2$ sells us $y_2=-0.5x_1+2x_2$ oranges. For instance, if we pay both 1 dollar, i.e. $x_1=x_2=1$, then we must get 1 pound of apples and 1.5 of oranges.

We can express this relation in the form of a matrix $P$:

 2   -1
-0.5  2 

Then the merchants produce this much apples and oranges if we pay them $x$ dollars: $$y=Px$$

This works exactly like a matrix by vector multiplication.

Now, let's say instead of buying the goods from these merchants separately, we declare that there are two spending bundles we utilize. We either pay both 0.71 dollars, or we pay $d_1$ 0.71 dollars and demand 0.71 dollars from $d_2$ back. Like in the initial case, we go to a bazaar and spend $z_1$ on the bundle one and $z_2$ on the bundle 2.

So, let's look at an example where we spend just $z_1=2$ on bundle 1. In this case, the first merchant gets $x_1=1$ dollars, and the second merchant gets the same $x_2=1$. Hence, we must get the same amounts of produce like in the example above, aren't we?

Maybe, maybe not. You noticed that $P$ matrix is not diagonal. This indicates that for some reason how much one merchant charges for his produce depends also on how much we paid the other merchant. They must get an idea of how much pay them, maybe through rumors? In this case, if we start buying in bundles they'll know for sure how much we pay each of them, because we declare our bundles to the bazaar. In this case, how do we know that the $P$ matrix should stay the same?

Maybe with full information of our payments on the market the pricing formulas would change too! This will change our matrix $P$, and there's no way to say how exactly.

This is where we enter tensors. Essentially, with tensors we say that the calculations do not change when we start trading in bundles instead of directly with each merchant. That's the constraint, that will impose transformation rules on $P$, which we'll call a tensor.

Particularly we may notice that we have an orthonormal basis $\bar d_1,\bar d_2$, where $d_i$ means a payment of 1 dollar to a merchant $i$ and nothing to the other. We may also notice that the bundles also form an orthonormal basis $\bar d_1',\bar d_2'$, which is also a simple rotation of the first basis by 45 degrees counterclockwise. It's also a PC decomposition of the first basis. hence, we are saying that switching to the bundles is simple a change of coordinates, and it should not change the calculations. Note, that this is an outside constraint that we imposed on the model. It didn't come from pure math properties of matrices.

Now, our shopping can be expressed as a vector $x=x_1 \bar d_1+x_2\bar d_2$. The vectors are tensors too, BTW. The tensor is interesting: it can be represented as $$P=\sum_{ij}p_{ij}\bar d_i\bar d_j$$, and the groceries as $y=y_1 \bar d_1+y_2 \bar d_2$. With groceries $y_i$ means pound of produce from the merchant $i$, not the dollars paid.

Now, when we changed the coordinates to bundles the tensor equation stays the same: $$y=Pz$$

That's nice, but the payment vectors are now in the different basis: $$z=z_1 \bar d_1'+z_2\bar d_2'$$, while we may keep the produce vectors in the old basis $y=y_1 \bar d_1+y_2 \bar d_2$. The tensor changes too:$$P=\sum_{ij}p_{ij}'\bar d_i'\bar d_j'$$. It's easy to derive how the tensor must be transformed, it's going to be $PA$, where the rotation matrix is defined as $\bar d'=A\bar d$. In our case it's the coefficient of the bundle.

We can work out the formulas for tensor transformation, and they'll yield the same result as in the examples with $x_1=x_2=1$ and $z_1=0.71,z_2=0$.

  • 2
    $\begingroup$ I got confused around here: So, let's look at an example where we spend just z1=1.42 on bundle 1. In this case, the first merchant gets x1=1 dollars, and the second merchant gets the same x2=1. Earlier you say that first bundle is that we pay both 0.71 dollars. So spending 1.42 on the first bundle should get 0.71 each and not 1, no? $\endgroup$
    – amoeba
    Commented Feb 25, 2016 at 10:32
  • $\begingroup$ @ameba, the idea's that a bundle 1 is $\bar d_1/ \sqrt 2+\bar d_2/ \sqrt 2$, so with $\sqrt 2$ bundle 1 you get $\bar d_1+\bar d_2$, i.e. 1\$ each $\endgroup$
    – Aksakal
    Commented Feb 25, 2016 at 14:44
  • 2
    $\begingroup$ @Aksakal, I know this discussion is quite old, but I don't get that either (although I was really trying to). Where does that idea that a bundle 1 is $\bar d_1/ \sqrt 2+\bar d_2/ \sqrt 2$ come from? Could you elaborate? How is that when you pay 1.42 for the bundle both merchants get 1? $\endgroup$
    – Matek
    Commented Sep 14, 2016 at 8:16
  • $\begingroup$ @Aksakal This is great, thanks! I think you have a typo on the very last line, where you say x1 = x2 = 1 (correct) and z1 = 0.71, z2 = 0. Presuming I understood everything correctly, z1 should be 1.42 (or 1.41, which is slightly closer to 2^0.5). $\endgroup$ Commented Aug 3, 2017 at 2:14
  • $\begingroup$ @Aksakal from "Now, our shopping can be expressed as a vector..." your post is a bit unclear. Would you mind explaining it a bit more detail please? What's the $P$ notation you're using? Could this simply be explained in terms of change of basis? With your notation, it seems like $A$ would be the matrix for change of basis from $d'$ to $d$, correct? I.e. to get the "transformation" w.r.t bundle basis, we compose the transformation w.r.t. starting basis together with the change of basis matrix, correct? $\endgroup$
    – Jake1234
    Commented Sep 26, 2020 at 21:12

As someone who studies and builds neural networks and has repeatedly asked this question, I've come to the conclusion that we borrow useful aspects of tensor notation simply because they make derivation a lot easier and keep our gradients in their native shapes. The tensor chain rule is one of the most elegant derivation tools I have ever seen. Further tensor notations encourage computationally efficient simplifications that are simply nightmarish to find when using common extended versions of vector calculus.

In Vector/Matrix calculus for instance there are 4 types of matrix products (Hadamard, Kronecker, Ordinary, and Elementwise) but in tensor calculus there is only one type of multiplication yet it covers all matrix multiplications and more. If you want to be generous, interpret tensor to mean multi-dimensional array that we intend to use tensor based calculus to find derivatives for, not that the objects we are manipulating are tensors.

In all honesty we probably call our multi-dimensional arrays tensors because most machine learning experts don't care that much about adhering to the definitions of high level math or physics. The reality is we are just borrowing well developed Einstein Summation Conventions and Calculi which are typically used when describing tensors and don't want to say Einstein summation convention based calculus over and over again. Maybe one day we might develop a new set of notations and conventions that steal only what they need from tensor calculus specifically for analyzing neural networks, but as a young field that takes time.

  • $\begingroup$ Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own answers. $\endgroup$ Commented Jul 4, 2017 at 1:17

Now I actually agree with most of the content of the other answers. But I'm going to play Devil's advocate on one point. Again, it will be free flowing, so apologies...

Google announced a program called Tensor Flow for deep learning. This made me wonder what was 'tensor' about deep learning, as I couldn't make the connection to the definitions I'd seen.

enter image description here

Deep learning models are all about transformation of elements from one space to another. E.g. if we consider two layers of some network you might write co-ordinate $i$ of a transformed variable $y$ as a nonlinear function of the previous layer, using the fancy summation notation:

$y_i = \sigma(\beta_i^j x_j)$

Now the idea is to chain together a bunch of such transformations in order to arrive at a useful representation of the original co-ordinates. So, for example, after the last transformation of an image a simple logistic regression will produce excellent classification accuracy; whereas on the raw image it would definitely not.

Now, the thing that seems to have been lost from sight is the invariance properties sought in a proper tensor. Particularly when the dimensions of transformed variables may be different from layer to layer. [E.g. some of the stuff I've seen on tensors makes no sense for non square Jacobians - I may be lacking some methods]

What has been retained is the notion of transformations of variables, and that certain representations of a vector may be more useful than others for particular tasks. Analogy being whether it makes more sense to tackle a problem in Cartesian or polar co-ordinates.

EDIT in response to @Aksakal:

The vector can't be perfectly preserved because of the changes in the numbers of coordinates. However, in some sense at least the useful information may be preserved under transformation. For example with PCA we may drop a co-ordinate, so we can't invert the transformation but the dimensionality reduction may be useful nonetheless. If all the successive transformations were invertible, you could map back from the penultimate layer to input space. As it is, I've only seen probabilistic models which enable that (RBMs) by sampling.

  • 1
    $\begingroup$ In the context of neural networks I had always assumed tensors were acting just as multidimensional arrays. Can you elaborate on how the invariance properties are aiding classification/representation? $\endgroup$
    – Y. S.
    Commented Feb 25, 2016 at 16:02
  • $\begingroup$ Maybe I wasn't clear above, but it seems to me - if the interpretation is correct - the goal of invariant properties has been dropped. What seems to have been kept is the idea of variable transformations. $\endgroup$ Commented Feb 25, 2016 at 20:02
  • $\begingroup$ @conjectures, if you have a vector $\bar r$ in cartesian coordinates, then convert it to polar coordinates, the vector stays the same, i.e. it still point from the same point in the same direction. Are you saying that in machine learning the coordinate transformation changes the initial vector? $\endgroup$
    – Aksakal
    Commented Feb 25, 2016 at 20:18
  • $\begingroup$ but isn't that a property of the transformation more than the tensor? At least with linear and element-wise type transformations, which seem more popular in neural nets, they are equally present with vectors and matrices; what are the added benefits of the tensors? $\endgroup$
    – Y. S.
    Commented Feb 26, 2016 at 9:47
  • 1
    $\begingroup$ @conjectures, PCA is just a rotation and projection. It's like rotating N-dimensional space to PC basis, then projecting to sub-space. Tensors are used in similar situations in physics, e.g. when looking at forces on the surfaces inside bodies etc. $\endgroup$
    – Aksakal
    Commented Feb 26, 2016 at 12:32

Here is a lightly edited (for context) excerpt from Non-Negative Tensor Factorization with Applications to Statistics and Computer Vision, A. Shashua and T. Hazan which gets to the heart of why at least some people are fascinated with tensors.

Any n-dimensional problem can be represented in two dimensional form by concatenating dimensions. Thus for example, the problem of finding a non-negative low rank decomposition of a set of images is a 3-NTF (Non-negative Tensor Factorization), with the images forming the slices of a 3D cube, but can also be represented as an NMF (Non-negative Matrix Factorization) problem by vectorizing the images (images forming columns of a matrix).

There are two reasons why a matrix representation of a collection of images would not be appropriate:

  1. Spatial redundancy (pixels, not necessarily neighboring, having similar values) is lost in the vectorization thus we would expect a less efficient factorization, and
  2. An NMF decomposition is not unique therefore even if there exists a generative model (of local parts) the NMF would not necessarily move in that direction, which has been verified empirically by Chu, M., Diele, F., Plemmons, R., & Ragni, S. "Optimality, computation and interpretation of nonnegative matrix factorizations" SIAM Journal on Matrix Analysis, 2004. For example, invariant parts on the image set would tend to form ghosts in all the factors and contaminate the sparsity effect. An NTF is almost always unique thus we would expect the NTF scheme to move towards the generative model, and specifically not be influenced by invariant parts.

[EDIT] Just discovered the book by Peter McCullagh, Tensor Methods in Statistics.

Tensors display interest properties in unknown mixture identification in a signal (or an image), especially around the notion of the Canonical Polyadic (CP) tensor decomposition, see for instance Tensors: a Brief Introduction, P. Comon, 2014. The field is known under the name "blind source separation (BSS)":

Tensor decompositions are at the core of many Blind Source Separation (BSS) algorithms, either explicitly or implicitly. In particular, the Canonical Polyadic (CP) tensor decomposition plays a central role in identification of underdetermined mixtures. Despite some similarities, CP and Singular Value Decomposition (SVD) are quite different. More generally, tensors and matrices enjoy different properties, as pointed out in this brief introduction.

Some uniqueness results have been derived for third-order tensors recently: On the uniqueness of the canonical polyadic decomposition of third-order tensors (part 1, part 2), I. Domanov et al., 2013.

Tensor decompositions are nodaways often connected to sparse decompositions, for instance by imposing structure on the decomposition factors (orthogonality, Vandermonde, Hankel), and low rank, to accommodate with non-uniqueness.

With an increasing need for incomplete data analysis and determination of complex measurements from sensors arrays, tensors are increasingly used for matrix completion, latent variable analysis and source separation.

Additional note: apparently, the Canonical Polyadic decomposition is also equivalent to Waring decomposition of a homogeneous polynomial as a sum of powers of linear forms, with applications in system identification (block structured, parallel Wiener-Hammerstein or nonlinear state-space models).


May I respecfully recommend my book: Kroonenberg, P.M. Applied Multiway Data Analysis and Smilde et al. Multiway Analysis. Applications in the Chemical Sciences (both Wiley). Of interest may also be my article: Kroonenberg, P.M. (2014). History of multiway component analysis and three-way correspondence analysis. In Blasius, J. and Greenacre, M.J. (Eds.). Visualization and verbalization of data (pp. 77–94). New York: Chapman & Hall/CRC. ISBN 9781466589803.

These references talk about multway data rather than tensors, but refer to the same research area.


What the term tensor means depends on the context it is used in:

Field Meaning
Machine learning Multi-dimensional array (usually numeric)1 2
Maths an algebraic object describing a (multilinear) relationship between sets of algebraic objects

The machine learning term is inspired by the fact that in a fixed basis/coordinate system, a tensor can be expressed as a multi-dimensional array. However, there are alternative representations, depending on the sub-field, with varying degrees of abstraction.

  1. Tensors, also known as multidimensional arrays, are generalizations of matrices to higher orders and are useful data representation architectures.

  2. Tensor

    The primary data structure in TensorFlow programs. Tensors are N-dimensional (where N could be very large) data structures, most commonly scalars, vectors, or matrices. The elements of a Tensor can hold integer, floating-point, or string values.


It is true that people in Machine Learning do not view tensors with the same care as mathematicians and physicians. Here is a paper that may clarify this discrepancy: Comon P., "Tensors: a brief introduction" IEEE Sig. Proc. Magazine, 31, May 2014

  • 7
    $\begingroup$ Is the distinction between a tensor in mathematics/physics and a tensor in machine learning really one of "care"? It seems that machine learning folks use "tensor" as a generic term for arrays of numbers (scalar, vector, matrix and arrays with 3 or more axes, e.g. in TensorFlow), while "tensor" in a math/physics context has a different meaning. Suggesting that the question is about "care" is, I think, to mischaracterize the usage as "incorrect" in the machine learning capacity, when in fact the machine learning context has no intention of precisely replicating the math/physics usage. $\endgroup$
    – Sycorax
    Commented Jan 16, 2018 at 16:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.