So I was thinking, if I have a set of features (let's say $(X_1, X_2, X_3)$) that basically describe the same overarching feature $Y$, and can somehow be mapped $(X_1, X_2, X_3) \rightarrow Y$. In addition, there exists a forth feature $X_4$ which is not part of this mapping but still contains meaningful data and is conceptually is on the same level as $Y$, meaning $X_4 \rightarrow X_4$.

To relate this to a more tangible example, $(Height, Muscle Density, Weight) \rightarrow Gender$ and the additional independent feature $HairColor$. Furthermore, let's assume that $HairColor$ can somehow be meaningful represented as a continuous variable, all features are normalized properly and I don't want to apply factor analysis.

Now, I want to explore this dataset with 4 features $(X_1, X_2, X_3, X_4)$, meaning I want to cluster. The majority of cluster algorithms (if not even all) intrinsically rely on distance metrics. This leads me to my first question:

1) The features mapping to $Y$ will account for the majority of distances between observations within my dataset and the distances of $X_4$ between observations will become negligible. This probably results in two clusters (or in our contemporary age maybe more), namely male and female. Information about the hair color will get lost. Is this interpretation/claim (at least in some circumstances) correct?

2) And if so, how would one express/prove this in a formal way?

Thank you in advance!


That is just the usual feature weighting problem.

Instead of assuming hair color to be continuous, consider a variable such as income. It's probably on the range of 1:100000, and thus will dominate the other three variables...

If you want to study this more formally, begin with a model where the differences in each variable is composed of signal $w_i$ and error $\epsilon_i$ and then study how much effect each variable has. Use Manhattan distance, as this is easiest to handle.

  • $\begingroup$ Isn´t that the reason why you standardize your variables? My question was targeting every $L^q$ distance. My idea currently is to use cosine similarity instead of $L^q$ metrics. $\endgroup$ – Raphael Prager Mar 21 '19 at 13:11
  • $\begingroup$ Standardization is a hack. It creates as many problems as it solves when you have correlated variables or useless variables. Which you can see from the very same considerations... If you have a duplicate variable and standardize everything, that variable gets twice the weight, for example. $\endgroup$ – Has QUIT--Anony-Mousse Mar 21 '19 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.