I have a dataset of text documents and I'm calculating pairwise cosine distances among them. For each document I have a bag of words vector, a vector built from entities extracted from the document, and the time the document was created. Each type of vector is of different dimensions.

I want the distance calculation to use all of these features, but such that the distance is biased towards the concepts vector. I have been calculating the distances for each type of vector independently then combining them as a weighted average, but I suspect there's a more correct approach.


Just do it.

Cosine boils down to computing scalar products (with each other, and each vector with itself when computing the magnitude),

$$\langle a,b\rangle = \sum_i a_ib_i$$

which can trivially be weighted

$$\langle a,b\rangle_\Omega = \sum_i \omega_ia_ib_i$$

Choose $\omega_i$ such that each feature set has the same sum of weights. For example, choose $\omega_i=1/d$ where $d$ is the number of features in this features space. You can move the $\omega_i$ term out if it is constant for all attributes. You will likely end up with something as easy as this:

$$\langle a,b\rangle = \frac{1}{|F_1|}\langle F_1(a), F_1(b)\rangle + \frac{1}{|F_2|}\langle F_2(a), F_2(b)\rangle + \ldots$$

which is the same as scaling/weighting each attribute in each of the feature spaces by $1/|F_i|$ where $|F_i|$ is the dimensionality of the features space.

You can further simplify the process if you first normalize all vectors to unit length (in each feature space separately). Then $||a||=||b||=1$, and cosine does become the scalar product: $$\cos = \frac{\langle a,b\rangle}{||a||\cdot ||b||} = \langle \frac{a}{||a||},\frac{b}{||b||}\rangle =_{\text{if }||a||=||b||=1} \langle a,b\rangle$$ As this equation shows (using bilinearity of the scalar product), you do get the same result.

Let $F_i^\prime(a)=F_i(a)/||F_i(a)||$ be the data normalized to $L_2$ norm 1. Then you can just choose

$$\text{sim}(a,b) = \frac{1}{k}\sum_{i=1}^k \langle F_i^\prime(a),F_i^\prime(b)\rangle$$

If $F^\prime(a) = \frac{1}{k} \mathop{concat}_{i=1}^k F_i^\prime(a)$ is the concatenation of the L2 unit normalized vectors; divided by $k$, then this is the classic scalar product/cosine:

$$\text{sim}(a,b) = \langle F^\prime(a), F^\prime(b)\rangle$$

In other words, normalize data in each feature space independently to unit length, then each features space gets the same weight.

  • $\begingroup$ Cosine boils down to the scalar product. The scalar product, devoid of the vectors' magnitudes, since $a \cdot b= |a||b|cos$. $\endgroup$ – ttnphns Sep 17 '14 at 9:06
  • $\begingroup$ Magnitude is computed via the scalar product with itself - so if you have understood it for the scala product once, you can use it there, too. $\endgroup$ – Anony-Mousse Sep 17 '14 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.