I have a data-set consisting of N p-dimensional observations (all quantitative variables). I want to apply a hierarchical clustering algorithm to those data. As explained on page 505 in Elements of Statistical Learning, when using weighted average to combine the distances of the individual variables, it is often desirable (I have found no clues to the contrary in my scenario), to set the weights for each variable such that all variables have equal influence on the distance of the observations (formulas can be found in the book). The problem is, that I use scipy.spatial.distance for calculating the distances, and this does no let me specify any weights. My question is, if I standardize my observations (multiply each dimension with 1/average of that dimension), does that solve the problem?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
I'd just say: be careful with that. Standarization is needed only when some variable(s) dominates the dissimilarity score just because it is expressed in "smaller units"; let's say that you have a variable that is truly equal for all elements, but there is some, very small variability due to the measurement error. Now if you'd normalize this value, you'll make those errors an important factor for clustering.