# How can I calculate cosine distance with multiple feature vectors and weigh them?

I have a dataset of text documents and I'm calculating pairwise cosine distances among them. For each document I have a bag of words vector, a vector built from entities extracted from the document, and the time the document was created. Each type of vector is of different dimensions.

I want the distance calculation to use all of these features, but such that the distance is biased towards the concepts vector. I have been calculating the distances for each type of vector independently then combining them as a weighted average, but I suspect there's a more correct approach.

Just do it.

Cosine boils down to computing scalar products (with each other, and each vector with itself when computing the magnitude),

$$\langle a,b\rangle = \sum_i a_ib_i$$

which can trivially be weighted

$$\langle a,b\rangle_\Omega = \sum_i \omega_ia_ib_i$$

Choose $\omega_i$ such that each feature set has the same sum of weights. For example, choose $\omega_i=1/d$ where $d$ is the number of features in this features space. You can move the $\omega_i$ term out if it is constant for all attributes. You will likely end up with something as easy as this:

$$\langle a,b\rangle = \frac{1}{|F_1|}\langle F_1(a), F_1(b)\rangle + \frac{1}{|F_2|}\langle F_2(a), F_2(b)\rangle + \ldots$$

which is the same as scaling/weighting each attribute in each of the feature spaces by $1/|F_i|$ where $|F_i|$ is the dimensionality of the features space.

You can further simplify the process if you first normalize all vectors to unit length (in each feature space separately). Then $||a||=||b||=1$, and cosine does become the scalar product: $$\cos = \frac{\langle a,b\rangle}{||a||\cdot ||b||} = \langle \frac{a}{||a||},\frac{b}{||b||}\rangle =_{\text{if }||a||=||b||=1} \langle a,b\rangle$$ As this equation shows (using bilinearity of the scalar product), you do get the same result.

Let $F_i^\prime(a)=F_i(a)/||F_i(a)||$ be the data normalized to $L_2$ norm 1. Then you can just choose

$$\text{sim}(a,b) = \frac{1}{k}\sum_{i=1}^k \langle F_i^\prime(a),F_i^\prime(b)\rangle$$

If $F^\prime(a) = \frac{1}{k} \mathop{concat}_{i=1}^k F_i^\prime(a)$ is the concatenation of the L2 unit normalized vectors; divided by $k$, then this is the classic scalar product/cosine:

$$\text{sim}(a,b) = \langle F^\prime(a), F^\prime(b)\rangle$$

In other words, normalize data in each feature space independently to unit length, then each features space gets the same weight.

• Cosine boils down to the scalar product. The scalar product, devoid of the vectors' magnitudes, since $a \cdot b= |a||b|cos$. – ttnphns Sep 17 '14 at 9:06
• Magnitude is computed via the scalar product with itself - so if you have understood it for the scala product once, you can use it there, too. – Anony-Mousse Sep 17 '14 at 9:14