0
$\begingroup$

I've done a clustering and I think that my results are too good to be trusted. Here is my pipeline:

  1. Inputs: a dataset of 208 images, distributed into 2 classes (99 and 109 images in each class).
  2. Extraction of 500 features for each image.
  3. Center (average 0) and reduce (std 1) each feature.
  4. Features selection with weka, using PCA+Ranker, which gave me a subset of 24 features.
  5. Clustering using 3 different algorithms: EM, K-means, X-means

The K-means and X-means provide perfect results (100% matching, according to the confusion matrix), and the EM has 80% accuracy.

I feel that my results are somewhere corrupted and consequently that I am doing something wrong.

  • Is there something obviously wrong in my pipeline?
  • Would you have any idea about how to confirm or contradict my results?
Improve this question
$\endgroup$
9
  • $\begingroup$ What do you mean by "perfect results"? Do you mean that every image in a given cluster was of the same class? $\endgroup$ – jld Mar 8 '16 at 1:35
  • $\begingroup$ yes, exactly! All my images belonging to the same class, were clustered into the same cluster $\endgroup$ – FiReTiTi Mar 8 '16 at 1:42
  • 2
    $\begingroup$ Use cross-validation. $\endgroup$ – air Mar 8 '16 at 8:48
  • 1
    $\begingroup$ @Wayne, sure, but I think the poster has never even heard of cross-validation, since (s)he is looking at performance only on the training set, so my comment was just meant to point at the right direction and not to give a full fledged answer. Even "wrongly" applied cross-validation would be better than what (s)he is doing right now... $\endgroup$ – air Mar 8 '16 at 18:27
  • 1
    $\begingroup$ @FiReTiTi: Cross-Validation is a technique of dividing your data (repeatedly) into a training and a testing partition. So you would take your training data, do feature extraction, do feature selection, etc, then run your clustering on the testing data -- data that none of your steps up to that point have used in any way. $\endgroup$ – Wayne Mar 8 '16 at 18:47
3
$\begingroup$

From your comment, you tell the PCA+Ranker your two classes. It then picks variables that will most reliably split your samples into those two classes. You then run clustering with those variables and it breaks those samples into two classes.

What you've essentially done is turned this into a supervised problem, then tested on your training data. In this case, you're essentially looking at in-sample accuracy, which can easily be 100%.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, it's what I feared. Except a cross validation, is there a better approach? $\endgroup$ – FiReTiTi Mar 8 '16 at 18:05
  • $\begingroup$ Cross-validation is not magic. It's better than train/validate (held-out), not because it's actually different but because it uses your data more efficiently and provides distributions rather than point numbers -- both of which are good. And where you use cross-validation (or a hold-out validation set) matters: in your case you have to use it at a high level: around your PCA+Ranker and your clustering, not just on your clustering. (I.e. loop steps 2-5 of your algorithm for each fold.) $\endgroup$ – Wayne Mar 8 '16 at 18:19
  • $\begingroup$ Ok, it's exactly what I expected to do. But it takes much more time. $\endgroup$ – FiReTiTi Mar 8 '16 at 18:29
2
$\begingroup$

If I were a reviewer, I would reject such results as extremely implausible.

Confusion matrixes are also not commonly used with clustering, because of the correspondence problem: how do you know which cluster is which class?

Instead, ARI is the most common evaluation measure.

To check your pipeline: really (really!) remove the labels until you do the evaluation. Do not let the labels leak into the clustering in any way. You may want to try random labels for comparison. If you still get 100% on random labels, you have a problem.

As is, I'd say the chance that you have an error somewhere is 100%.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's the reason why I was suspicious and I didn't publish the results. I keep the labels, but they are not used during the clustering, so I don't loose the correspondance. What is ARI? $\endgroup$ – FiReTiTi Mar 8 '16 at 18:08
  • 1
    $\begingroup$ ARI=Adjusted Rand Index. Double check that the labels aren't in your clustering, and also that PCA+Ranker doesn't overfit on the labels (e.g. PCA must not have the labels either, and by using them during feature selection you are effectively supervised). $\endgroup$ – Has QUIT--Anony-Mousse Mar 8 '16 at 20:31
1
$\begingroup$

With only 208 cases and 500 features, you've way, way overfit the model. What you're witnessing is exhaustion of the degrees of freedom so, of course, the fit is perfect.

Improve this answer
$\endgroup$
3
  • $\begingroup$ As I said, I've reduced the number of features to 24. $\endgroup$ – FiReTiTi Mar 8 '16 at 18:09
  • $\begingroup$ Still sounds like you've exhausted the degrees of freedom. What are the features like? How are they composed? They're probably spanning every aspect of the matrix space $\endgroup$ – Mike Hunter Mar 8 '16 at 18:35
  • $\begingroup$ That's statistical methods like haralick, Run Length Matrix, Size Zone Matrix, Local Binary Patterns. $\endgroup$ – FiReTiTi Mar 8 '16 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.