0
$\begingroup$

The issue of feature scaling and weighting for cluster formation has been widely discussed in several books and papers as well as several questions (e.g. here ). To my understanting, variable range is the one to be considered as the weight of a variable that affects cluster formation, so as variables with large ranges dominate the ones with smaller ranges, particularly when using Euclidean distances. This is stated in Hastie et al (2011) and Kaufman & Rousseeuw (1990) among others. To overcome this problem, a form of feature scaling is suggested in order to balance the variables, so each variable can play and equal role in cluster formation. Min-max normalisation seems to be the most widely used scaling method in the literature. However, since clustering is problem-dependent, variables considered to be more relevant in separating groups, should be assigned a higher influence factor (Hastie et al 2011). That is, in other words, more relevant variables should be assigned a different weight and consequently have larger range.

Considering the above, and to fit it to my problem, I consider one of my variables to be more relevant and consequently I want to assign a higher weight to it, so that it will influence cluster formation more than other variables. However, scalling all other variables to the same range and leaving the important one as it is, results into clusters dominated by the important variable only, since it has very large range. The most rational solution then, would be to reduce the range of the "important variable" as well, so as it would still influnce cluster formation, but to a degree where other variables will be considered as well. That is, for example, bring all variables to the same range (e.g. -1 to 1) and the important variable to a different range (e.g. 0 to 100) or bring all variables to the same range and then add or multiply the important variable with a constant.

Hence, my question is: Is it "correct" to scale variables to different ranges before applying clustering? More important, if it is correct, what is the proper methodology to do it? Is it sensible to scale the variables using "random" ranges/weights?

Based on the literature, it is indeed correct to use different ranges/weight for variables as mentioned. However, I have been unable to identify any applications that make clear this methodology. Any references would me much appreciated.

Some references for clustering methodology, but not applications can be found here:

Hastie, Tibshirani, Friedman. 2011. The Elements of Statistical Learning

Kaufman L, Rousseeuw P. 1990. Finding Groups in Data - An Introduction to Cluster Analysis

Greenacre M, Primicerio R. 2015. Multivariate Analysis of Ecological Data

$\endgroup$
  • $\begingroup$ Also, have you looked in some local threads about standardizing, such as stats.stackexchange.com/q/21222/3277 or stats.stackexchange.com/q/372521/3277 ? $\endgroup$ – ttnphns Mar 25 at 15:04
  • $\begingroup$ bring all variables to the same range and the important variable to a different range OR bring all variables to the same range and then add or multiply the important variable by a constant. The effect, whether this or that way will bring higher "importance" to a variable at clustering depends on the nature of the cluster analysis you are using. Is it distance based? Which distance? One has to know how the algorithm works in order to answer. $\endgroup$ – ttnphns Mar 25 at 15:10
  • $\begingroup$ @ttnphns thank you for the links. I have concluded that I need to do normalisation. Based on your second comment: I am applying K-means with Euclidean distance. My question actually is "can I apply random weights? if yes, how? How is this justifiable?" $\endgroup$ – Alex Mar 25 at 15:50
  • $\begingroup$ If you are going to do classic k-means that is implicitly based on euclidean didtances then weifgting is most simple problem - see my last comment in one of the linked questions $\endgroup$ – ttnphns Mar 25 at 16:54
  • $\begingroup$ Sorry, not there, but see the comment in the Q you linked to yourself, here it is stats.stackexchange.com/q/77850/3277 $\endgroup$ – ttnphns Mar 25 at 17:00
0
$\begingroup$

Min-max scaling as well as standardization often is not sufficient. Non-linear scaling may often be necessary to achieve the desired effects.

There is no "correct" way. Variable importance in an unsupervised context is a parameter that you have to choose.

The easiest way often (ignoring non-linear transformations for now) is to standardize variables, and then increase the weight of the variable you consider more important to 2x,3x, etc. until the results make most sense for you.

Minmax scaling is usually much worse than standardization, as it depends on the two most extreme values only, which tend to be outliers.

$\endgroup$
  • $\begingroup$ Thank you so much. This answer makes everything clearer. Do you have any references for this? $\endgroup$ – Alex Mar 25 at 20:29
  • $\begingroup$ No. Because what would be publishable here? Essentially I'm saying there is no theory applicable here, and you have to experiment. You probably write a paper on that. $\endgroup$ – Anony-Mousse Mar 25 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.