Standardizing some features in K-Means

Question

I have 21 features in my dataset, some features are more important than others. As a fact I know, if I don't standardize (mean=0, SD=1) any features, then features with low variance will have slightly more effect in determining the final clusters. My doubt is: I want my dataset to reflect that some features are important and some are relatively less important, before I call the k-means. Is there any way to implement this using R studio?

Answer 1

If you are using K-means, it is essential to make sure different features are in same scale. This is because K-means use Euclidean distance and the scale of the feature have great impact on such measure.

Consider a toy example where you have $2$ data points with 2 features for each point. The tow points are $p_1=(0.01,1000)$ and $p_2=(1,3000)$.

The distance between $p_1$ and $p_2$ is roughly $2000$, which is the distance on your second feature. Note that in the first feature two data points has $100$ times differences, but it got "ignored" by Euclidean distance.

To extend the toy example, please see following simulation for a more extreme on feature scales with visualization. Following figure shows two data sets and two results from K-means clustering. The synthesized data is evenly distributed in a square, so, the desired output would like a grid, as shown in the left plot.

• The first data set x has 2 features, both of them are in 0,1
• The second data set x2 has 2 features, one is in the scale of $10^7$ another feature is in the scale of $10^{-7}$.

The results shows if you do not make 2 features in same scale, the feature with large scale will be dominant and the other feature become useless.

set.seed(0)
par(mfrow = c(1,2))
x = matrix(runif(5000), ncol = 2)
cluster = kmeans(x, 10, nstart = 20)
plot(x, col = cluster$cluster, pch = 20)

x2 = x
x2[,1] = x2[,1]*1e7
x2[,2] = x2[,2]*1e-7
cluster2 = kmeans(x2, 10, nstart = 20)
plot(x2, col = cluster2$cluster, pch = 20)

---

In addition to the scale, Euclidean distance (K-means) may suffer from "curse of dimensionality", see this post

Answer 2

If your sample size isn't incredibly large, you could also check out using gower distance along with k-medoids clustering. Gower distance allows you to set user-specified weights for each variable when calculating distance between observations, so you would be able to apply more weight to what you consider to be "important" variables. One thing to note is that Gower uses manhattan distance for continuous variables instead of Euclidean, which may not be what you want. At any rate, see below for a quick example using the iris dataset in R.

library(cluster)

my_data <- iris[, 1:4]

# Weight Sepal.Length more heavily than the other three variables

dist_mat <- daisy(x = my_data,
                  metric = "gower",
                  weights = c(0.4, 0.2, 0.2, 0.2))

# Use k-medoids to extract three clusters

pam_fit <- pam(x = dist_mat,
               diss = TRUE,
               k = 3)

Answer 3

You would need to clarify what "important" means to you. One fundamental requirement before you run cluster analysis is to choose the set of variables you want to run your analysis on.

If you know apriori which variables are "important" to you, then cluster your data using only them and leave out the irrelevant variables.

As to your point of variables with low variance having an impact that is true, especially in sparse datasets. On the flip side, if you do not standardize, variables that have higher variance will disproportionately contribute to the distance measure.