2
$\begingroup$

I have a dataset which contains 10000 examples. Each example has 100 dimensions. These dimensions have the same scale.

I clustered all examples using their 100-dimensional vectors and drew the elbow chart to find the appropriate number of clusters. enter image description here

It seems that it is appropriate to make the number of clusters equal to 3.

Then, I wanted to plot the dataset. So I reduced the 100 dimensions to 2 dimensions using Principal Component Analysis (PCA). enter image description here

I actually do not see any clue from the 2-D plot that the dataset can be separated into 3 clusters... It seems that all examples huddle together.

I also labelled three clusters by color and tried to see how k-means separates the data. enter image description here

I feel that the clustering is kind of "rigid" in the 2D space.

So my questions are:

1) I trust the result of the elbow plot moreat this moment. So how does PCA help people to understand whether the dataset can be separable?

2) If PCA shows that a dataset cannot be separated well in 2D space, does it mean that the data still can be well separated in the original high dimension space?

============ EDIT: I Have also tried Multidimensional scaling (MDS).

enter image description here

It seems that it does not look much better than PCA... Three clusters are not very "natural".

========== EDIT 1/26:

@ttnphns @Anony-Mousse @usεr11852 Thanks for your comments. You guy are right. I just find that there are a lot of all-zero vectors in my dataset. Since they are meaningless in my application. So now I removed them, reduce dimensions by PCA, and plot it again.

Here is the new elbow chart. No obvious elbow shows up. I think this is normal for real-life datasets (according to Andrew Ng). enter image description here

Here is the scatter plot for 4 clusters and 5 clusters: enter image description here

enter image description here

I think It looks more reasonable now.

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ Why have you decided the data have 3 clusters? It sooner have no clusters. Did you erroneously expect that the withinSS scree plot would be straight line under no clusters? Did you try a more fine criterion (such a Calinski-Haracasz) instead of withinSS? Even if you had some good clusters in 100 original dimensions they are not guarantee to show through in 2 or 3 first PCs. $\endgroup$
    – ttnphns
    Jan 23, 2016 at 21:35
  • 3
    $\begingroup$ Anyway, in spite the k-means didn't serve you clear clusters it yet has segmented the dull triangular shape into a fancy coralfish. $\endgroup$
    – ttnphns
    Jan 23, 2016 at 21:35
  • 2
    $\begingroup$ how much of the original information do your two components carry? Just express the sum of two eigen values as a fraction of the sum of all eigen values -- this can give a clue. If the fraction is too small, then your two components do not represent the original dataset that well. So the 2D plots you have created may not mean much. $\endgroup$
    – Minaj
    Jan 23, 2016 at 21:36
  • 2
    $\begingroup$ I agree with ttnphns: there are no clusters here, the curve is falling of as expected. Draw 10000 samples from a single 100 dimensional Gaussian i.i.d. and the curve will look similar, too. Your visualizations also indicate that this data set does not cluster (maybe it is not preprocessed well - how do your individual dimensions look like?) $\endgroup$
    – Has QUIT--Anony-Mousse
    Jan 24, 2016 at 1:19
  • 1
    $\begingroup$ +1 @ttnphns comments. Just to add some obvious things: If anything you seem to be having outliers; there might be a story there. They seem so massive at time that probably they are corrupted observations, check these samples. Also there must be a decent story as why you get that sharp triangular shape in your PC1 vs PC2 projections (the fish's nose). This is clearly a non-standard diffusion pattern. $\endgroup$
    – usεr11852
    Jan 24, 2016 at 6:09

1 Answer 1

Reset to default
4
$\begingroup$

Such patterns can arise when you have inappropriate data for PCA.

For example, when your data is sparse and discrete.

pdata = np.random.poisson(.005, 1000*1000).reshape((1000,1000))
p = sklearn.decomposition.TruncatedSVD(2).fit_transform(pdata)
scatter(p[:,0], p[:,1])

enter image description here

The fish-like shape arises because:

  • the data is very skewed (most values are 0 due to sparsity)
  • there are no negative values

In such cases, PCA will often not work very well.

PCA is about maximizing variance in the first components. However, variance is a quantity that makes most sense for continuous and non-sparse data. PCA (usually, if you use the original version) centers your data; but already this operation only makes sense when the data generation process can be assumed to be translation invariant (e.g. Normal distributions don't change their shape when you translate the mean).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. I do find that there are a lot of all-zero vectors in my dataset. They are meaningless in my application. So now I removed them. I just edit the question to show the current findings. $\endgroup$
    – Munichong
    Jan 27, 2016 at 2:15
  • $\begingroup$ Sparsity is not only about all-zero vectors, but about zero-values in general. Your new visualizations look much better - but: there are clearly no clusters in the data set judging from these plots. There is some non-linear correlation there (probably some angle?) but no clusters, sorry. $\endgroup$
    – Has QUIT--Anony-Mousse
    Jan 27, 2016 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.