# How to standardize a data-set

I have a data-set consisting of N p-dimensional observations (all quantitative variables). I want to apply a hierarchical clustering algorithm to those data. As explained on page 505 in Elements of Statistical Learning, when using weighted average to combine the distances of the individual variables, it is often desirable (I have found no clues to the contrary in my scenario), to set the weights for each variable such that all variables have equal influence on the distance of the observations (formulas can be found in the book). The problem is, that I use scipy.spatial.distance for calculating the distances, and this does no let me specify any weights. My question is, if I standardize my observations (multiply each dimension with 1/average of that dimension), does that solve the problem?

-
Ideally, the question should be self-contained. Could you please give the relevant formulas as well? By the way, I may not have understood your question properly but won't euclidean distance give the same weight to all the variables? That metric seems to be one of the distance measures offered by the library you are using. – user28 Sep 3 '10 at 10:42
instead of "(multiply each dimension with 1/average of that dimension)" don't you mean to multiply by the reciprocal of the standard deviation of the corresponding coordinates? This is not a bad choice but it's sensitive to outliers. Rescaling by a resistant measure of spread, such as the IRQ, would protect you from outliers and still accomplish what you intend. – whuber Sep 3 '10 at 14:19
@Srikant Vadali: I wanted to post the formulas, but I couldn't find a way to embed them here, and ASCII-formulas would have made things worse instead of better. Also, the book I referenced is freely available as an ebook, so everyone can look it up. – Björn Pollex Sep 3 '10 at 15:07
Check out meta.stats.stackexchange.com/q/386/930 for LaTeX typesetting – chl Sep 3 '10 at 15:09