0
$\begingroup$

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section discussion frequency, one-hot, and TF-IDF encodings as a way to encode words into vector spaces, the authors state that

Unfortunately, these vectorization methods produce document vectors with non-negative elements, which means we won’t be able to compare documents that don’t share terms (because two vectors with a cosine distance of 1 will be considered far apart, even if they are semantically similar).

I'm unclear why this is exactly the case- why are non-negative elements not useful for comparing documents that don't share terms? Wouldn't you just have a larger encoding space, with lots more 0s? Furthermore, if you really wanted to use negative elements, couldn't you standardize your encoding to have mean of 0?

I'm also unclear how this portion relates to non-negative elements at all. (I understand that a cosine distance of 1 is the maximum distance):

because two vectors with a cosine distance of 1 will be considered far apart, even if they are semantically similar

Could someone point me in the right direction of what the authors' intended meaning of this sentence is?

$\endgroup$
4
$\begingroup$

I'm unclear why this is exactly the case- why are non-negative elements not useful for comparing documents that don't share terms?

Just because two documents don't share terms doesn't mean they're dissimilar. In other words, Bag of Words model works good if it says something is similar (of course you can run into problems with similar words with different meaning), but it does a worse job telling something is dissimilar.

An example for the following statement:

because two vectors with a cosine distance of 1 will be considered far apart, even if they are semantically similar

Let

s1 = "A young woman owns a cat"
s2 = "A girl has a kitten"

Then, after deleting 'a', which would be considered as a stopword by most tools for text preprocessing, these sentences will have biggest possible cosine distance - they do not share a word. But they have very similar meaning.

Furthermore, if you really wanted to use negative elements, couldn't you standardize your encoding to have mean of 0?

That would be a different encoding. It would also destroy useful properties - bag of words encodings have mostly zeros, so they can be represented using sparse vectors - if you were to represent couple thousand examples that have tens of thousands of dimensions you'd need much more memory.

There is other method that sort of does what you described, but it also uses dimensionality reduction - it's called Latent Semantic Analysis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.