I know that this might sound like a really dum or naive question but I believe it's not (I hope). I've noticed that a default used to be to have one hot vectors to encode words in a vocabulary. But doing that fixes the vocabulary vector size to a fix size. Which can be bad if we want a AI agent to be able to incorporate new concepts as it is learning more. While just indexing with real numbers doesn't have this problem at all. Why don't we use some representation like this for increasing vocabulary or concept knowledge base problems (that DON'T fix the vector size)?
I guess I am interested in a flexible representation suitable for learning that allows for growing concept sizes. I guess the only thing I am aware of that does this are (dense) embeddings in real vector spaces.