Embedding data into a larger dimension space

Question

Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has many discrete features that doesn't make sense to turn each of them to a new dimension. Embedding makes it possible to embed data into a lower dimensional space that dimensions do not correspond to the features. Depending on the situation embedding can be trained separately or simultaneously with the rest of the model. My question is: are there possible benefits for embedding into a space of larger or same dimension?

The reason I am asking the question is that, I am reading a paper which claims one of their innovations is that they are embedding into a latent space and train the embedding together with the rest of the network. Lowering dimensionality is stated as one of the benefits of this action. But in their code implementation they have examples that is embedding a 5 dimensional space into 24 dimensions.

Answer 1

As Forrest mentioned embedding data into a higher dimension (sometimes called basis expansion) is a common method which allows a linear classifier to observe a non-linear input space. Examples are using the RBF kernel with an SVM or polynomial expansion with linear regression.

However basis expansion isn't always beneficial. For a classification task if your data is already linearly separable the only thing basis expansion will do is increase model complexity and training time. For a regression task basis expansion can lead to overfitting the training data if regularization is not imposed, effectively fitting the model to noise in the training set.

And certain models don't benefit from basis expansion at all. For example deep neural networks will have no noticeable benefit from embedding the inputs into a higher dimension space. This is because the (non-linear) hidden layers of a network can be seen as performing basis expansion where the specific higher dimension embedding is learnt by the network during training.

So as a summary, the merit of embedding data in a higher dimension depends on what model you are using and properties of your data. Embedding data in a higher dimension prior to using a linear model is common to attempt to introduce linear separability. Embedding data in a higher dimension is also something that occurs implicitly in some models such as SVMs using the kernel trick or neural networks with non-linear activations.

Answer 2

are there possible benefits for embedding into a space of larger or same dimension?

In Vector Symbolic Architectures (also known as Hyperdimensional Computing) this is essential. VSAs use algebraic operations (sum, product, permutation) on vectors to encode information in their directions. VSAs can be used to represent discrete data structures such as graphs as well as continuous properties (like brightness) as attributes of the discrete structure. They rely on the properties of high-dimensional spaces to work. You're unlikely to come across a VSA working in a less than 500-d space. 10,000-d is typical.

Low dimensional input values are projected up to the VSA dimensionality choosing a method that preserves the input properties of interest. Note that although individual component (like a pixel brightness) are being massively expanded in dimensionality, the dimensionality of the entire structure being encoded (say a 500 x 500 x RGB image) may be being reduced in dimensionality to a single VSA encoding (i.e. one vector represents the entire image).

For an introduction, see http://www.rctn.org/vs265/kanerva09-hyperdimensional.pdf

they have examples that is embedding a 5 dimensional space into 24 dimensions

In such low dimensional spaces they are very unlikely to be using VSA. But the point remains, also stated in other answers, expanding the basis might be useful - it depends what you are trying to do.