Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a data-set consisting of N p-dimensional observations (all quantitative variables). I want to apply a hierarchical clustering algorithm to those data. As explained on page 505 in Elements of Statistical Learning, when using weighted average to combine the distances of the individual variables, it is often desirable (I have found no clues to the contrary in my scenario), to set the weights for each variable such that all variables have equal influence on the distance of the observations (formulas can be found in the book). The problem is, that I use scipy.spatial.distance for calculating the distances, and this does no let me specify any weights. My question is, if I standardize my observations (multiply each dimension with 1/average of that dimension), does that solve the problem?

share|improve this question
    
Ideally, the question should be self-contained. Could you please give the relevant formulas as well? By the way, I may not have understood your question properly but won't euclidean distance give the same weight to all the variables? That metric seems to be one of the distance measures offered by the library you are using. –  user28 Sep 3 '10 at 10:42
    
instead of "(multiply each dimension with 1/average of that dimension)" don't you mean to multiply by the reciprocal of the standard deviation of the corresponding coordinates? This is not a bad choice but it's sensitive to outliers. Rescaling by a resistant measure of spread, such as the IRQ, would protect you from outliers and still accomplish what you intend. –  whuber Sep 3 '10 at 14:19
    
@Srikant Vadali: I wanted to post the formulas, but I couldn't find a way to embed them here, and ASCII-formulas would have made things worse instead of better. Also, the book I referenced is freely available as an ebook, so everyone can look it up. –  Björn Pollex Sep 3 '10 at 15:07
    
Check out meta.stats.stackexchange.com/q/386/930 for LaTeX typesetting –  chl Sep 3 '10 at 15:09
add comment

1 Answer 1

up vote 2 down vote accepted

I'd just say: be careful with that. Standarization is needed only when some variable(s) dominates the dissimilarity score just because it is expressed in "smaller units"; let's say that you have a variable that is truly equal for all elements, but there is some, very small variability due to the measurement error. Now if you'd normalize this value, you'll make those errors an important factor for clustering.

share|improve this answer
    
Thanks for your answer. I just found a useful paper on that issue: dx.doi.org/10.1007/BF01897163. It compares the usefulness of different normalization approaches with each other and with no normalization. –  Björn Pollex Sep 3 '10 at 15:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.