I have a dataset where each row represents a sample and each sample is described by its chemical composition. You can see the 10 first rows of the dataset in figure 1.

Image1 - dataset Figure 1 - Each row represent a sample and each sample is decomposed into the 17 different chemical compounds and the total (all values are given in percentage)

First I found the correlation between the samples and made the correlation matrix shown in figure 2.

Correlation matrix

But what I really want to cluster the chemical compounds that are more likely to be found together in a sample.


You seem to look for cluster analysis. Cluster analysis groups data according to some distance measure and correlation may well be the basis for your distance measure(*). As you have not mentioned any rules of how well samples should correlate to be toghether in one group, hiearchical cluster analysis might be in order: It will reveal visually the structure of how many groups do form depending on how you set a cutoff.

(*) https://www.datanovia.com/en/lessons/clustering-distance-measures/ writes

Correlation-based distance considers two objects to be similar if their features are highly correlated, even though the observed values may be far apart in terms of Euclidean distance. The distance between two objects is 0 when they are perfectly correlated. Pearson’s correlation is quite sensitive to outliers. [...]

If we want to identify clusters of observations with the same overall profiles regardless of their magnitudes, then we should go with correlation-based distance as a dissimilarity measure

| cite | improve this answer | |
  • $\begingroup$ Do you think I could use something like K-means Clustering? $\endgroup$ – bjornsing Apr 3 at 14:01
  • 1
    $\begingroup$ Absolutely, if you have a sensible idea of the size of $k$. Turn your correlation matrix into a correlation based distance matrix (see link given in the answer) and run the $k$-means algorithm. Also read my link about standardisation. $\endgroup$ – Bernhard Apr 3 at 14:05
  • 1
    $\begingroup$ Won't k-means or other clustering methods cluster the samples rather than the chemical components ? It seems to me that you are trying to cluster features (the chemical components) and not the data (the samples), or am I missing something ? $\endgroup$ – Pohoua Apr 3 at 14:08
  • 1
    $\begingroup$ Ok, I misunderstood that, but the structure of the problem does not change. You will need to compute a correlation based distance matrix of the chemical components, then, before you do either hierarchical or $k$-means clustering. $\endgroup$ – Bernhard Apr 3 at 14:13
  • 1
    $\begingroup$ Point 7 in the "Notes" paragraph looks promising. Now look for a clustering function that can take the output of pdist(X, 'correlation') as it's input. I am no Python user so I cannot give you more hints as to which combination of functions/methods will do the trick but I'd expect some hints to worthwhile functions (including the heatmaps with dendrograms in fig 3) in mycarta.wordpress.com/2019/04/10/… $\endgroup$ – Bernhard Apr 3 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.