5
$\begingroup$

Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has many discrete features that doesn't make sense to turn each of them to a new dimension. Embedding makes it possible to embed data into a lower dimensional space that dimensions do not correspond to the features. Depending on the situation embedding can be trained separately or simultaneously with the rest of the model. My question is: are there possible benefits for embedding into a space of larger or same dimension?

The reason I am asking the question is that, I am reading a paper which claims one of their innovations is that they are embedding into a latent space and train the embedding together with the rest of the network. Lowering dimensionality is stated as one of the benefits of this action. But in their code implementation they have examples that is embedding a 5 dimensional space into 24 dimensions.

$\endgroup$
1
  • 3
    $\begingroup$ The kernel trick is a common example of embedding data in a higher dimensional space. It is common to do this in support vector machines as it can help to improve linear separability of the data. $\endgroup$
    – Forrest
    Jun 4, 2021 at 3:40

3 Answers 3

9
$\begingroup$

As Forrest mentioned embedding data into a higher dimension (sometimes called basis expansion) is a common method which allows a linear classifier to observe a non-linear input space. Examples are using the RBF kernel with an SVM or polynomial expansion with linear regression.

However basis expansion isn't always beneficial. For a classification task if your data is already linearly separable the only thing basis expansion will do is increase model complexity and training time. For a regression task basis expansion can lead to overfitting the training data if regularization is not imposed, effectively fitting the model to noise in the training set.

And certain models don't benefit from basis expansion at all. For example deep neural networks will have no noticeable benefit from embedding the inputs into a higher dimension space. This is because the (non-linear) hidden layers of a network can be seen as performing basis expansion where the specific higher dimension embedding is learnt by the network during training.

So as a summary, the merit of embedding data in a higher dimension depends on what model you are using and properties of your data. Embedding data in a higher dimension prior to using a linear model is common to attempt to introduce linear separability. Embedding data in a higher dimension is also something that occurs implicitly in some models such as SVMs using the kernel trick or neural networks with non-linear activations.

$\endgroup$
1
$\begingroup$

are there possible benefits for embedding into a space of larger or same dimension?

In Vector Symbolic Architectures (also known as Hyperdimensional Computing) this is essential. VSAs use algebraic operations (sum, product, permutation) on vectors to encode information in their directions. VSAs can be used to represent discrete data structures such as graphs as well as continuous properties (like brightness) as attributes of the discrete structure. They rely on the properties of high-dimensional spaces to work. You're unlikely to come across a VSA working in a less than 500-d space. 10,000-d is typical.

Low dimensional input values are projected up to the VSA dimensionality choosing a method that preserves the input properties of interest. Note that although individual component (like a pixel brightness) are being massively expanded in dimensionality, the dimensionality of the entire structure being encoded (say a 500 x 500 x RGB image) may be being reduced in dimensionality to a single VSA encoding (i.e. one vector represents the entire image).

For an introduction, see http://www.rctn.org/vs265/kanerva09-hyperdimensional.pdf

they have examples that is embedding a 5 dimensional space into 24 dimensions

In such low dimensional spaces they are very unlikely to be using VSA. But the point remains, also stated in other answers, expanding the basis might be useful - it depends what you are trying to do.

$\endgroup$
0
$\begingroup$

I suggest that the datapoints of the 5-dimensional dataset are first classified in 13 types of classes(orbits) where these types of classes(orbits) have the following cardinalities: 1,10,32,40,80,160,240,320,480,640,960,1920,3840. These 13 types of classes are generated by the intrinsic properties of the automorphism of Z^5, where Z^5 is the 5-D integer lattice. For other dimensions please have a look at the OEIS sequence A270950. Use the infinity norm of the datapoint to create a 2D-table where the columns are the cardinalities and the rows the infinity norm. Each cell in the table represents the number of classes (orbits) with the chosen cardinality and containing datapoints having the same infinity norm. These tables are mathematical invariants and have only to be calculated once and can be used for any 5-D dataset.

$\endgroup$
2
  • $\begingroup$ Like your other post, the existence or nature of any connection with the question is quite baffling. I think we can say with confidence that your "mathematical invariants" have no bearing on the statistical problems addressed by the kinds of embeddings this question is concerned with. $\endgroup$
    – whuber
    Jun 25, 2022 at 1:22
  • $\begingroup$ I wanted to point out that you don't need to embed a 5-dimensional data set in a 24-dimensional data set for making the "mathematical classification" easier. The first step should be to stay in the 5-dimensional dataset and "color" the data points with one of the 13 types that can exist in a 5D dataset. A 24D dataset has 1751 "colors" making the classification task more time consuming. $\endgroup$ Jun 26, 2022 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.