Comparing k-means results to original data: how to interpret the resulting plots?

Question

I'm running k-means on my dataset that can be found here that has 7 classes.

I plotted the ggpairs for the dataset and then took k-means and plotted ggpairs again in an effort to see what effect it has.

The two plots are below, but I'm not sure how to interpret them.

How should I compare the original plot to the one against k-means (7 clusters) plot? What sort of information can I extract out of them?

Plotting actual data (7 classes)

Plotting k-means with 7 clusters

Code for the plots

#original data plot
wine <- read.csv("wine_nocolor.csv")
library(GGally)
wine1 <- wine[2:13]
wine1$q <- factor(wine1$q)
ggpairs(wine1, columns=1:11,
        colour='q',lower=list(continuous='points'),
        axisLabels='none',
        upper=list(continuous='blank'), title="Actual Wine Quality")
#k-means with 7 clusters
set.seed(1234)
wine1$Cluster7 <- factor(kmeans(wine[1:11], 7)$cluster)
ggpairs(wine1, columns=1:11, colour='Cluster7', lower=list(continuous='points'), axisLabels='none', upper=list(continuous='blank'), title="k-means with 7 clusters")

Answer 1

How should I compare the original plot to the one against k-means (7 clusters) plot? What sort of information can I extract out of them?

You can extract the information that the k-means results are garbage.

If you look at your second figure, you will see that the clustering is based almost exclusively on the tsd variable: your 7 clusters are basically 7 ranges of tsd values.

The reason this happens, is that tsd has largest numerical values. K-means operates on Euclidean distances and is highly sensitive to normalization. Having one variable that is on a much larger scale than all the others, can completely bias the clustering.

I think we can safely conclude that this particular analysis is useless.

---

Let me add that as you have the quality of each wine (your 7 "classes", which are in fact levels of an ordinal variable) explicitly available in your dataset, you should probably be doing some sort of regression (such as e.g. ordinal logistic regression), and not clustering. I cannot see any possible value of trying to cluster this dataset.

Answer 2

The scale for each variable is quite different, and this affects the cluster analysis, eg

> summary(wine1[,1:5])

      fa               va               ca               rs               ch         
 Min.   : 3.800   Min.   :0.0800   Min.   :0.0000   Min.   : 0.600   Min.   :0.00900  
 1st Qu.: 6.400   1st Qu.:0.2300   1st Qu.:0.2500   1st Qu.: 1.800   1st Qu.:0.03800  
 Median : 7.000   Median :0.2900   Median :0.3100   Median : 3.000   Median :0.04700  
 Mean   : 7.215   Mean   :0.3397   Mean   :0.3186   Mean   : 5.445   Mean   :0.05603  
 3rd Qu.: 7.700   3rd Qu.:0.4000   3rd Qu.:0.3900   3rd Qu.: 8.100   3rd Qu.:0.06500  
 Max.   :15.900   Max.   :1.5800   Max.   :1.6600   Max.   :65.800   Max.   :0.61100

Also a couple of the variables have skewed distributions which makes it hard to separate using clustering. You should transform the skewed variables and standardize the variables, and then run k-means for a better clustering.

Here is the revised code:

wine <- read.csv("data/wine_nocolor.csv", stringsAsFactors=FALSE)
library(GGally)
library(reshape)
library(ggplot2)
library(dplyr)
wine1 <- wine[,2:13]
wine1$q <- factor(wine1$q)

wine1$va <- log10(wine1$va)
wine1$rs <- wine1$rs^(0.25)
wine1$ch <- log10(wine1$ch)
wine1$fsd <- wine1$fsd^(0.25)
wine1$s <- log10(wine1$s)
set.seed(1234)
wine1$Cluster7 <- kmeans(rescaler(wine1[1:11]), 7)$cluster
wine1$Cluster7 <- factor(wine1$Cluster7, levels=1:7, 
                         labels=letters[1:7])
table(wine1$q, wine1$Cluster7)

wine1.m <- melt(wine1[,c(1:11,13)])
qplot(Cluster7, value, data=wine1.m, geom="boxplot", colour=Cluster7) + 
  facet_wrap(~variable, scales="free_y") + theme(legend.position="bottom")

The results look much better now. You can see that the clustering has divided the data into reasonable chunks, eg cluster c has high values on fa, ... The data has no natural clusters, so k-means is simply breaking the data into roughly equal chunks. The clusters do not match the original classes, but the original classes don't separate the data at all.

Answer 3

Focus on the tsd column only for now. As you can see from the stripes only this attribute has an effect on your clustering. You simply cut your data set into slices based on tsd.

This is because you did not preprocess your data well. The attributes have different scale. And some appear to be nonlinear. k-means only works if your attributes are comparable and linear. I.e. a difference of 1 must be the same in every attribute and at every location (0 to 1 must bear the same significance as 1 million to 1 million + 1)

You could compute for example the ARI (adjusted Rand index) to compare the clustering result to the known classes. But it will score really badly, because you skipped the first step: preprocessing your data.

Answer 4

I basically agree with amoeba's comment but it doesn't offer much in the way of a solution.

What I would do is first drop q from the inputs of the clustering. Secondly, you should standardize your variables to mean zero and unit variance. Now if you try your k-means again, it will likely not be as heavily dependent on tsd. However, you may also want to try other algorithms such as Gaussian Mixture Models or Affinity Propagation. Afterwards you can summarize your clusters by your ordinal variable.

But this plotting approach is not the best for clustering, in my arrogant opinion. I usually plot the data on the first two or three PCA coordinates. Doing the PCA can also tell you a lot about the variables. For instance, if you had done it on this dataset ahead of time you would have seen that 1) the first PC dimension dominates the clustering and 2) tsd is very highly correlated with the first PC.

Clustering is never an exercise in perfection, but keep at it and see if you get a result that is useful.

Answer 5

I'll first discuss comparison methods, but I would also like to make a point:

In order to compare the k-means to the actual data, you should compare the distributions, since k-means is ultimately giving you an estimate of the distribution (see below). To compare the distributions, you should use the notion of "divergence" between the actual and predicted distributions. In this case, you can use the KL-divergence (aka the relative entropy) between the distributions for each class.

Since your distributions are discrete, you can use the following equation.

Given distributions $P$ and $Q$, the KL-divergence $D_{KL}(P||Q)$ is defined as follows $$ D_{KL}(P||Q) = \sum_i P(x_i)\log_2\frac{P(x_i)}{Q(x_i)} $$

Intuitively, this can be thought of (loosely) as a metric of the information lost when using $Q$ to try and predict $P$. In this case, $P$ is the actual distributions and $Q$ is the k-means distribution. In other words, how much information is lost when you try to use k-means to predict the actual data. The remarkable thing about the KL-divergence is that it gives you an unbiased estimate of the divergence. It can also be extended to multiple dimensions with the Renyi Divergence, but I wouldn't worry about that for your data.

Note that the base of the log is arbitrary - I used base 2 since these ideas come form information theory.

You can compare these plots by plotting the KL-divergence for each class for each plot.

---

Honestly, I don't think k-means is appropriate here. k-means makes a few important assumptions, which helps explain why you cannot extract the correct distribution in this case:

1. k-means assumes that the variance of the distribution of each class is spherical (for example, gaussian)
2. all classes have the same variance
3. the prior probability of each cluster is equal (equal number of observations)

You can see clearly that the first and second statements do not hold, so k-means will fail. You could, on the other hand, try single linkage hierarchical clustering. Here is a brilliant article about what this means in practice: http://varianceexplained.org/r/kmeans-free-lunch/

Let me know if you have any questions and I can elaborate.

Answer 6

Since your question was “to see what effect it has” or interpret k-means vs. raw data plots, rather than whether this was the best way to analyze or visualize the data, here’s my try: k-means clustering hints at stronger trends between e.g. fa and va, ca, p in the last-to-be-mapped pink cluster, which may benefit from further examination. If you make density plots for each cluster, other such trends may become evident, if the goal is visual trend scanning.

k-means clustering seems to be based almost exclusively on tsd. However, it seems almost too good to be true, so it is more likely to be an a priori known grouping.

Answer 7

This is a little vague, because k-means is meant to cluster, rather than to compare clusters to known groups. You might want to look into another technique. If you are still hooked on k-means clustering, you could always perform some sort of ordination, such as nMDS or RDA, and plot points to visualize both a priori and a posteriori (k-means) groups, as a start.

Another approach, which might be valuable to you, is to use LDA to construct a statistical model of your groups. With this technique you can see how well LDA predicts your a priori groups, and what variables contribute most to the formation of those groups. Unlike k-means, it is not so important that you standardize your data before hand (in response to amoeba's interpretation to your k-means results).

Good luck!

---

As an example, see the following:

library(MASS)
wine.lda <- lda(as.factor(q)~fa+va+ca+rs+ch+fsd+tsd+d+p+s+a, wine_nocolor)

plot(wine.lda)

wine.predict <- predict(wine.lda, newdata = wine_nocolor[2:12])

table(wine.predict$class, wine_nocolor$q) 
sum(diag(prop.table(table(wine.predict$class, wine_nocolor$q) )))

I got an accuracy of about 54%. You can look at the summary of the LDA results to pick out the major players. I believe in this case it is "d"... possibly due to very low variance in group 9. As opposed to what I said earlier, column scaling the data might give you nicer looking eigenvectors, however it won't change the model's accuracy.