2
$\begingroup$

Assume that you having $N$ clusters. Each cluster have multiple classes. So we know the class ID for every major clusters, but not the class ID for the data points inside the major clusters.

Each colour is its own class ID. E.g red can be class ID 1 and blue can be class ID 2. Assume that you are using Support Vector Machine to classify each major clusters. But in this case, SVM cannot classify each data point.

How should this be solved If I got $X$ number of points and I don't know which class they belong to, but I know which major cluster they fit in.

enter image description here

I found the major clusters with K-means clustering and then I'm using SVM with a linear kernel to find the mathematical expression how to classify each data point into the major clusters, but not succeed to find the class ID of the individual data points.

Do you have any suggestion?

$\endgroup$
1
  • $\begingroup$ If it's correct that the information being clustered into major clusters is categorical, discrete or nominal, which it sounds like it is, then k-means is NOT the appropriate algorithm. K-means is intended for use with continuously scaled information, ideally at the interval scale level. A more appropriate routine for categorical information or mixtures of scales is latent class clustering as originally developed by academics like Clifford Clogg, James Coleman, William Dillon and James Heckman. If the data is massive, then see Dunson's recent work on Bayesian Pyramids. $\endgroup$
    – user78229
    Nov 2, 2023 at 11:54

2 Answers 2

4
$\begingroup$

The trouble with your example is that these clusters are not particularly informative about the color you aim to predict. If one cluster were mostly blue, another mostly yelllow, etc, then the cluster (or the location in 2D space) would be informative about color/category. If your real work looks like the posted example, you probably do not have adequate data to make accurate predictions.

Distinct outcomes need to be different in the features for them to be distinguished from each other. Your example lacks this characteristic.

If there is a third feature (interpret it as being in and out of the image) and the colors in each cluster correspond to how far out of the image the points stick (might ot be the same for each color in each cluster), then knowing both the cluster (or just the $x$-$y$ coordinates) and the height might allow you to improve performance.

$\endgroup$
19
  • $\begingroup$ The last two images in my answer here seem related. $\endgroup$
    – Dave
    Nov 1, 2023 at 17:08
  • $\begingroup$ Thank you so much. You this problem is impossible to solve? Do you recommend for example UMAP or t-sne for reducing the dimension before linear SVM classification? $\endgroup$
    – euraad
    Nov 1, 2023 at 17:50
  • $\begingroup$ @euraad If nothing in the feature space differentiates one category from another, then there isn't much you can do. UMAP and t-SNE are nice tools for reducing the dimension to something that can be visualized, but if you just have two features like you have in your example, you can plot the entirety of the data set and examine, visually, if there is anything that distinguishes one color from any others. Your example seems to lack that. Perhaps your real data do not. // My answer at the link I gave is worth reading. This questions is, arguably, a duplicate in disguise. $\endgroup$
    – Dave
    Nov 1, 2023 at 17:53
  • $\begingroup$ Thank you so much. I will think a while which method I could use. My data is just typical image classification with object detection. $\endgroup$
    – euraad
    Nov 1, 2023 at 17:54
  • 1
    $\begingroup$ I understand that I need to find more dimension that can separate the data from each other. $\endgroup$
    – euraad
    Nov 1, 2023 at 18:15
4
$\begingroup$

This problem violates the so-called cluster hypothesis which states that points in the same cluster should generally belong to the same class. Here the clustering appears uninformative for determining the actual class of each individual. We have no useful measure of similarity of samples that actually belong to the same class, as the clustering seems to capture nothing useful about the classification under study. A clustering is useful when it groups items in some logical way according to similarities you actually care about. This is not a useful clustering in the context of this classification, as it does not group items of the same class together. It's a fundamental disconnect between the clustering and the classes, one is not useful for predicting the other.

$\endgroup$
6
  • $\begingroup$ So you are saying that this is not possible to make predictions if the same clusters shares the same classes? $\endgroup$
    – euraad
    Nov 1, 2023 at 17:38
  • 1
    $\begingroup$ @euraad It doesn't look like it, although there's no hard cutoff between "possible" and "not possible", just a decreasing ability to get a good result. The data you show looks very close to "not possible", as there is virtually no association whatsoever between the clustering and the classes you actually care about. It'd be like grouping people by the first letter of their last name, and trying to predict their favorite genre of movie - there just isn't any useful relationship between the two to leverage. $\endgroup$ Nov 1, 2023 at 17:45
  • 1
    $\begingroup$ @euraad You can always make predictions, but what you've shown in your example makes it seem like those predictions cannot consistently be accurate. For instance, if you know that a new point belongs to that cluster on the right, doesn't it seem like there is a one-in-seven chance that it belongs to each color group? That seems to be about all you can say. $\endgroup$
    – Dave
    Nov 1, 2023 at 17:48
  • $\begingroup$ I understand! Well, perhaps I need to add some dimension reduction. If you wonder what the data is, it's binary data e.g 0b110100101001001. I want to classify this. The data comes from ORB or SIFT classification for images. The most common tool is to use K-NN with sum(XOR(Class Image data, Unknown Image data)). If the sum is small, then the image belongs to a specific class. $\endgroup$
    – euraad
    Nov 1, 2023 at 17:52
  • 1
    $\begingroup$ @euraad You could possibly do very slightly better than random guessing by recognizing that being in the top two clusters gives a slightly increased probability of being in the brown class, while being in the bottom right cluster implies that it's not the yellow class. But with almost all classes observed in all clusters, even if the performance is "better than random", it's still likely in the range of "not useful". $\endgroup$ Nov 1, 2023 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.