# Tensors in neural network literature: what's the simplest definition out there?

In the neural network literature, often we encounter the word "tensor".

Is it different from a vector? And from a matrix? Have you got any specific example that clarifies its definition?

I'm a bit confused about its definition. Wikipedia doesn't help and sometimes I have the impression that its definition depends on the specific machine learning environment used (TensorFlow, Caffee, Theano).

For the purposes of data analysis, you can effectively consider them as arrays, possibly multidimensional. Thus they include scalars, vectors, matrices, and all higher order arrays.

The precise mathematical definition is more complicated. Basically the idea is that tensors transform multilinear functions to linear functions. See (1) or (2). (Multilinear functions are functions which are linear in each of their components, an example being the determinant considered as a function of column vectors.)

One consequence of this mathematical property defining tensors is that tensors transform nicely with respect to Jacobians, which encode transformations from one system of coordinates to another. This is why one often sees the definition of tensor as "an object which transforms in a certain way under changes of coordinates" in physics. See this video for example, or this one.

If we are dealing with sufficiently "nice" objects (all of the derivatives we would like to exist and well-defined are), then all of these ways of thinking about tensors are essentially equivalent. Note that the first way to think of tensors which I mentioned (multidimensional arrays) ignores the distinction between covariant and contravariant tensors. (The distinction is with regards to how their coefficients change under a change of basis of the underlying vector space, i.e. between row and column vectors essentially.) See these other StackExchange questions: (1) (2) (3) (4)

For a book used by researchers studying applications of tensors to neural networks (for example at Technion in Israel), there is Wolfgang Hackbusch's Tensor Spaces and Numerical Calculus. I have not read it yet myself, although some of the later chapters seem to use advanced mathematics.

• +1. Very related: stats.stackexchange.com/a/198395/28666. It seems to me that in machine learning, tensors are simply arrays and nothing more. In mathematics, tensors are understood differently; and usually the word "tensor" is used more narrowly than an "element of a tensor product". See linked answer. – amoeba Sep 4 '16 at 9:16

### Tensor = multi-dimensional array

In the machine learning literature, a tensor is simply a synonym for multi-dimensional array:

Tensors, also known as multidimensional arrays, are generalizations of matrices to higher orders and are useful data representation architectures.

Hence a 1.d tensor is a "vector/tuple", and a 2.d. tensor is a "matrix/2.d.array".

Theano vs TensorFlow vs Pytorch vs ...

In specific libraries the term may be restricted to numerical arrays:

Theano is a Python library that allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.

or those containing a broader range of data-types:

Tensor - The primary data structure in TensorFlow programs. Tensors are N-dimensional (where N could be very large) data structures, most commonly scalars, vectors, or matrices. The elements of a Tensor can hold integer, floating-point, or string values.

Etymology

Tensor has a more specific meaning in mathematics as an abstraction of a multilinear map between vector spaces, but given a fixed basis such maps can be represented as multidimensional arrays, and it is from this usage that the machine learning term gets its name.