3
$\begingroup$

Short question: is it meaningful to use tSNE ( http://homepage.tudelft.nl/19j49/t-SNE.html) to modify existing high-dimensional data using similarities in some low-dimensional vectors? In essence, that means applying tSNE in reverse direction ( but with constraint, that we are already given high-dimensional vectors and just wanted to "deform" them using the information in low-dimensional vectors ).

Long question: Here is my task : I have a list of word embeddings ( in word2vec sense ), and these vectors all are 64-dimensional.

On another hand, for each word a have another vector ( 15-dimensional ), and mapping is one-to-one ( i.e. each word have 64-dimensional embedding and 15-dimensional signature ). 15-dimensional vectors contain important additional information about words, which is impossible to take into account directly.

And I wanted to make these 64-dimensional vectors more similar by taking into account similarity between corresponding 15-dimensional vectors.

Any advices in this direction are highly appreciated. Thanks in advance!

$\endgroup$
0
$\begingroup$

You could append the 15 dimensional vector to the 64 dimensional vector, obtaining a 79 dimensional vector. You can then reduce the dimensionality by projecting on eigensubspaces.

In general however, there are infinitely many ways to do this, so you need to come up with a way some sort of objective criterion for evaluating the quality of your embedding.

word2vec attempts to maximize the predictive ability of the embedding in skip-grams. What you could do is re-train word2vec, but using the 15 dimensional vectors as a side information in the network to predict the missing word.

$\endgroup$
  • $\begingroup$ Thanks, @Arthur B., I was thinking about this already, but many problems are in this direction, and normalization issues are most important of them ( 15-dims vectors contain very large numbers ( hundred millions )), plus entire concatenation operation doesn't look theoretically sound. But if we "deform" vectors using some side information, this at least look understandable ( i.e. we optimize KL divergence, after all ): in essence, we are doing something like kernel trick, no? And no pain with normalization... $\endgroup$ – Vast Academician Aug 21 '15 at 13:42
  • $\begingroup$ Deform all you want, you still won't know by how much you should "deform" unless you tie this to some objective criterion. $\endgroup$ – Arthur B. Aug 21 '15 at 13:56
  • $\begingroup$ Sorry, I don't get your point. Why KL divergence ( which works in tSNE ) is not such a criterion to your opinion?! We just act like in basic t-SNE algorithm, but instead of initializing "learned" map with randoms, we initialize it with word2vec vectors. But t-SNE works if input dimensionality, say, 1000 dims and output is just 2d. Why will it break, if input is 15 dims and output is 64? $\endgroup$ – Vast Academician Aug 21 '15 at 14:06
  • $\begingroup$ You can do that, but it's no less arbitrary than concatenation. $\endgroup$ – Arthur B. Aug 21 '15 at 14:10
  • $\begingroup$ Ah, if I understood you right, you mean, that no operations on embedding ( be it concatenation, deformation, projection e t.c. ) are really meaningful because they don't add more information ( on contrary, retraining with side information is different, and thus more solid )? Yes? Honestly, I don't think, that whole idea of word2vec dimensions is solid enough :-) Because what these dimensions mean anyway? $\endgroup$ – Vast Academician Aug 21 '15 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.