Addressing "Unbalanced Features" or Feature Taxonomy for Nearest Neighbor / Similarity Calculations

Question

The main question is how to address an imbalance in representation of feature "sets" when calculating similarity. I'll motivate with an example scenario:

Suppose we have objects described by a binary feature vector. That binary feature vector contains ten bits:

• Eight bits to represent "color" categories, e.g. {red, orange, yellow, green, blue, indigo, violet, black}.
• Two bits to represent "texture" categories: e.g. {checker, stripe}

Note they are binary-encoded, and not one-hot-encoded. So each "bit" is treated independently. For example, an object can be characterized as both "red" and "yellow", or "checker and stripe".

Say we'd like to calculate nearest neighbors to one object, using the feature vector. There are various ways to do this, but let's consider Jaccard Index.

The Jaccard Index is computed with the Intersection-Over-Union. My concern is that we have a ten-bit feature vector, of which 8 out of the 10 bits represents a single feature. So even though we have two "features" (color and texture), the "color" feature dominates 80% (8/10) of the representation.

Is there a means of computing similarity or nearest-neighbors that manages the inbalance in representation of features, or incorporates this "taxonomy" naturally?

Answer 1

You can add weight to the features vectors to rebalance the similarity.

This will work with similarity measures that are based on numbers, and not bits - like cosine similarity (you can also define a variation on the jacard similarity that considers the weight of each bit)

technically, this means to normalise each "sub vector" and then concatenate them

Answer 2

Okay first why do you think this as an imbalance, if you consider theory of information gain and shanon entropy, you can see an object can be represented with 2 or more colour variables, so its not an imbalance actually it is the information that you get about an object, e.g. an apple can be represented as red and yellow or red alone or red yellow and green, to me its not an imbalance in features its the information and it can be attributed to colour alone or texture alone, any similarity or distance metrics will be based on this information, if you still think features are imbalanced try balancing them by adding more features related to texture.

Please discuss if i did not reach your point.