5
$\begingroup$

I want to cluster data that contains binary variables, exponentially distributed (power law) variables, and normally distributed variables. I'm considering preprocessing the data in the following way and wondering whether it's reasonable.

1) shift the binary variables so that they have mean zero. no rescaling.

2) standardize the normally distributed variables, but divide by twice the standard deviation rather than by once. this is based on Gelman, A. "Scaling regression inputs by dividing by two standard deviations. Statistics in Medicine 2008 27:2865-2837. [for clarification: their point is that binary variables can be interpreted directly as indicators and can be left unscaled. But in order for numerical inputs to be interpreted in the same way, they should be divided by twice the standard deviation -- that way, both numerical and binary variables have similar standard deviations (if the binary indicators are not strongly skewed; if they are, then there is not much improvement, but also no harm).]

3) take log(x+1) [or sqrt(x)] for any exponentially distributed variable x, and standardize in the above way.

Mainly, I'm worried about (3). I'm told that it's standard practice to simply standardize exponentially distributed variables. Is it standard practice because it's correct or because it's a good enough approximation? Also, would it be better to apply k-medoid clustering rather than k-means?

$\endgroup$

1 Answer 1

2
$\begingroup$

1.) Binary variables are a problem for many many algorithms. While mapping them to two numerical values $x_0,x_1\in\mathbb{R}$ works, the results are often "damaged" by not having any value inbetween; similar to the effects resulting from discrete values such as integers.

2.) Standardization is a best practise when dealing with different scales. Divding each attribute by two standard deviations instead of one standard deviation will not make any fundamental difference. It will just half all your distances.

3.) $\log(x+1)$ often works, but so does $\sqrt{x}$. Without theoretical support, either is just a guess to make things work. Standardization is still reasonable, as discussed above.

The key question you should be asking is how do I reasonably measure similarity? Don't start with how do I make k-means work on this data set. First find out what the problem is that you are trying to solve, then find the appropriate tools to do so. Don't let the tool define your problem.

$\endgroup$
5
  • $\begingroup$ Thanks for your response! I think I'm asking several questions here. On the reference mentioned above, their point is that binary variables can be interpreted directly as indicators and can be left unscaled. But in order for numerical inputs to be interpreted in the same way, they should be divided by twice the standard deviation -- that way, both numerical and binary variables have similar standard deviations (if the binary indicators are not strongly skewed; if they are, then there is not much improvement, but also no harm). $\endgroup$
    – rebka
    Jun 4, 2014 at 19:41
  • $\begingroup$ As to not letting the tool define my problem, I guess, indirectly, I'm asking what is a reasonable measure of similarity? I'll clarify above, but for variables with power law distributions...let's take for example number of days spent on x by various people: I care more about the difference between people who spent 5 and 10 days, than about those who spent 105 and 110 days. This is why log or square root seem more reasonable. But then again, I can avoid this issue with medians. $\endgroup$
    – rebka
    Jun 4, 2014 at 19:43
  • $\begingroup$ They can be interpreted as indicators, but does this imply they should have weight 1? Are they are important as a difference of 1 in your other attributes? Similarity is use case dependant. 1 yard is not 1 gallon; log scale or not. You have to find a domain-specific tradeoff. $\endgroup$ Jun 4, 2014 at 19:54
  • $\begingroup$ "Are they are important as a difference of 1 in your other attributes?" - that's why we standardize at all, but I see your point on the arbitrariness, even in the case of a difference of 1 in one binary variable vs another. There's some model-driven vs data-driven issue here as well. Very deep. Thanks for this $\endgroup$
    – rebka
    Jun 4, 2014 at 20:11
  • $\begingroup$ Standardization is just a heuristic, and it may work well, or be completely useless. If you have 100 binary attributes, and they are random, then giving them each a weight of 1 (or 0.5, as the standard deviation is up to 0.5) will overshadow any signal you have in the other attributes. The best strategy is to learn which attributes are important, and to which degree. In particular when they are binary, discrete, or nominal. $\endgroup$ Jun 4, 2014 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.