The main question is how to address an imbalance in representation of feature "sets" when calculating similarity. I'll motivate with an example scenario:

Suppose we have objects described by a binary feature vector. That binary feature vector contains ten bits:

  • Eight bits to represent "color" categories, e.g. {red, orange, yellow, green, blue, indigo, violet, black}.
  • Two bits to represent "texture" categories: e.g. {checker, stripe}

Note they are binary-encoded, and not one-hot-encoded. So each "bit" is treated independently. For example, an object can be characterized as both "red" and "yellow", or "checker and stripe".

Say we'd like to calculate nearest neighbors to one object, using the feature vector. There are various ways to do this, but let's consider Jaccard Index.

The Jaccard Index is computed with the Intersection-Over-Union. My concern is that we have a ten-bit feature vector, of which 8 out of the 10 bits represents a single feature. So even though we have two "features" (color and texture), the "color" feature dominates 80% (8/10) of the representation.

Is there a means of computing similarity or nearest-neighbors that manages the inbalance in representation of features, or incorporates this "taxonomy" naturally?


2 Answers 2


You can add weight to the features vectors to rebalance the similarity.

This will work with similarity measures that are based on numbers, and not bits - like cosine similarity (you can also define a variation on the jacard similarity that considers the weight of each bit)

technically, this means to normalise each "sub vector" and then concatenate them


Okay first why do you think this as an imbalance, if you consider theory of information gain and shanon entropy, you can see an object can be represented with 2 or more colour variables, so its not an imbalance actually it is the information that you get about an object, e.g. an apple can be represented as red and yellow or red alone or red yellow and green, to me its not an imbalance in features its the information and it can be attributed to colour alone or texture alone, any similarity or distance metrics will be based on this information, if you still think features are imbalanced try balancing them by adding more features related to texture.

Please discuss if i did not reach your point.

  • 1
    $\begingroup$ Without additional domain info, it's unfair for me to say that any "bit of information" is more important than any other. I was trying to motivate a 2-layer taxonomy, where color and texture should have equal weight, regardless of the number of values they can take on. Let's assume that is the case. Now let's go to an extreme case where we assume color can take on 1000 different values. If computing Jaccard globally (now using 1002 bits), the "color" feature dominates. Is there a technique to maintain balance, if it's desired, within the taxonomy? $\endgroup$
    – Tom
    Commented Dec 17, 2018 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.