1
$\begingroup$

I know that this might sound like a really dum or naive question but I believe it's not (I hope). I've noticed that a default used to be to have one hot vectors to encode words in a vocabulary. But doing that fixes the vocabulary vector size to a fix size. Which can be bad if we want a AI agent to be able to incorporate new concepts as it is learning more. While just indexing with real numbers doesn't have this problem at all. Why don't we use some representation like this for increasing vocabulary or concept knowledge base problems (that DON'T fix the vector size)?

I guess I am interested in a flexible representation suitable for learning that allows for growing concept sizes. I guess the only thing I am aware of that does this are (dense) embeddings in real vector spaces.

$\endgroup$
1
$\begingroup$

I guess the only thing I am aware of that does this are (dense) embeddings in real vector spaces."

Exactly, they get the best of both worlds: A fixed vector length and an ability to represent as many concepts as desired. While indexing with discrete values for each word will loose the advantage of fixed vector lengths, which is useful for many things. In particular many ML models require fixed vector sizes.

While just indexing with real numbers doesn't have this problem at all.

Yes, the indexing scheme you propose has another problem: words can be close in meaning to two others words, even though those 2 words are not close. For example:

The word "mother" is close to both "queen" and "father", but "queen" and "father" are not (as) close to each other. If you use an indexing scheme, then the only way you can have the word "mother" be close to both "queen" and "father" at the same time is if "queen" and "father" are also close to each other.

A dense vector embedding would allow "mother" to be close to "father" along one dimension and close to "queen" along a different dimension, while still keeping "queen" and "father" far apart from each other.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.