How to normalize skewed data before clustering?

Question

My question deals with what is the right way to normalize my data. My data consists 6 features, all together representing a state in an environment for reinforcement learning. My goal is to cluster states with KMeans, so of course I need to normalize values first.

Below are histograms of the different features:

(X axis is feature value, Y axis is number of appearances. if you wonder where are features 3 and 5, they are kinda look like normal)

I wonder what is the right way to normalize each feature (or all of them together) to use kmeans.

I tried to apply f-score (all features toghther, not one by one) but I wonder if it is the right away. The minimum value I get is -4.6 (in feature 2, as you can see it has values around zero) with maximum of around ~1. Are there better suggestions what can I do?

Answer 1

There are several ways you can follow:

1. Box-cox transform
2. Log transform
3. Square root transform

For detailed explanation you can follow this link https://towardsdatascience.com/top-3-methods-for-handling-skewed-data-1334e0debf45

Answer 2

The k-means algorithm is not guaranteed to converge to a global optimum and is very initialization dependent; furthermore, its objective of squared Euclidean distance results in it minimizing intra-cluster variance. This means for k-means, you generally want your covariance matrix to be spherical; i.e. the covariance matrix is proportional to the identity matrix.

This means techniques like z-score scaling or range normalization on each feature can be useful. To see why, try coming up with a 2D example of where k-means changes depending on the initialization -- it's a common exam question that I've gotten that lends more insight into the behavior of k-means.

Obviously, it's hard (impossible, even) to visualize things in 6 dimensions, but if you can't plot out your data to analyze it yourself, I'd recommend asking yourself -- why k-means as opposed to any other clustering algorithm?

Here's a link with more info on the assumptions of k-means clustering that I find useful.