I'm trying to find any relationship/patterns between a large number of rows in a dataset (~2000) and I'm thinking of using a correlation heatmap. However, after transforming the df using df = df.T.corr() and only plotting the first 100 rows with seaborn, it already starts to look unreadable:enter image description here

Is there a clearer way to do this with a larger number of rows?

  • 2
    $\begingroup$ Sorting the correlation matrix may provide clusters of variables, see here for one description of how to sort them. $\endgroup$ – Andy W Jul 11 '16 at 12:17
  • 2
    $\begingroup$ Any Python based solutions? $\endgroup$ – user3508494 Jul 11 '16 at 13:18
  • $\begingroup$ I found sns.clustermap(df.T.corr(), metric='correlation', method='centroid') which might do the trick. $\endgroup$ – tmrlvi Nov 22 '17 at 15:26
  • $\begingroup$ Try to do some basic clustering before (with the kernel trick if necessary), then order your dataset with respect to the classes. In python, use scikit-learn's k-means, PCA or whatever clustering technique works with your data. $\endgroup$ – Romain Reboulleau Oct 4 '18 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.