Within a scatterplot of two continuous variables. I am looking for a statistical method to determine if a 3rd variable that has a limited number of categories/groups (3-8) is able to produce visually well-defined segments within the original scatter plot.

The following scatterplot would be a example of a good segmentation:

enter image description here

It is clear from this scatter plot that the different scatters are well grouped and are distinct from the other.

Compare that with this one:

enter image description here

This one doesn't have any distinct groups. It may be true that there is some significant correlation/association for all three, i.e., the greater the population, the greater the area, but there are no clear "segments" here.

Initially I have had the idea of measuring the average distance from centroid, and calculate the percentage of dots of a group that are not intersecting with other groups excluding outliers (e.g. IQR method). This could work, but how would I get statistics from it that would signify a good vs bad segmentation? And before I reinvent the wheel, I wanted to check if there are any known/used statistics already.

Or perhaps a test for unimodality of the residuals (dip test)? If it is not unimodal I could try to get the modes? Anyhow, any pointers to robust methods appreciated.

Addendum: I've applied Christian's Hennig's answer using a Calinski-Harabasz Index to dervive the stats. For reference, I published the results here (PDF). The plots are ordered by p values with the Null Hypothesis of independence. p<0.05 indicates a stronger evidence of visual well defined k clusters, which can be nicely seen within the plots.

  • 5
    $\begingroup$ Gaussian mixture modeling could solve your problem. $\endgroup$
    – Amin Shn
    Feb 7 at 15:11
  • $\begingroup$ @AminShn good suggestion if the groupings are not known. That's not OP's example however. To do GMM, you should have some reasonable knowledge of the number of modes and the proportion that each mixture component contains. Most people look at the scatter plot and guess, which is fine for an exploratory analysis. The EM algorithm does the rest. $\endgroup$
    – AdamO
    Feb 7 at 16:53
  • $\begingroup$ "is able to produce visually well-defined segments" So you want some sort of test that tells based on purely numbers, whether the plot is gonna look visually well-defined or not? If so, then what is a actually the visual cue that you are looking for. In the second plot you can also draw ellipses or some other sort of boundary and compare the groups. $\endgroup$ Feb 7 at 17:16
  • $\begingroup$ I believe that visual only makes it more confusing. I was starting to think that your goal was about human perception and generating a rule of thumb such as, "do not plot a line graph with more than 5 lines", to determine whether the graph is too cluttered... $\endgroup$ Feb 8 at 6:50
  • 1
    $\begingroup$ ... so what is your real goal? For your application, how would you describe segmentation in a practical sense. For instance a practical application could be: it is for the creation of a product portfolio, and you need to figure out whether one product could suit already many people versus two products. If you make two products you might get more sales, but it may be more costly to produce. Then you could compare the area of highest density regions for the individual populations and the population as a whole. $\endgroup$ Feb 8 at 6:59

4 Answers 4


In cluster analysis, the Silhouette coefficient (SC; or Average Silhouette Width) is a distance-based statistic that measures the quality of a clustering, i.e., to what extent the objects are closer to other objects in the same class than to the closest class to which they don't belong.

This can also be computed for situations as yours in which there is a given grouping; for these data probably the Euclidean distance makes sense.

One qualification is that clusterings found by a cluster analysis method (for with the Silhouette was originally meant) tend to be better separated than data from underlying groupings that have a fairly large variation. Therefore I'd recommend to contrast the SC obtained for your categories (which may look disappointingly low for people who know typical values in cluster analysis) with a permutation test approach, i.e., simulate 1000 (say) data sets where you randomly reshuffle the group labels, compute the SC for all of these, and have a look to what extent (measured in terms of standard deviations of the permutation results, say) the SC in your data is "significantly" larger.

The webpage also mentions a Simplified Silhouette that comes with less computational effort.

Sleeping over this, I realised that I should also mention another classical cluster validity index, the Calinski-Harabasz index (CH), In R here. It can once more be calibrated (or a statistical test be run) using the permutation principle. More than the SC, this is based on the standard statistics characterising the Gaussian distribution, namely mean vector and sums of squares, so will be appropriate for within-group distributions that are not too far from the Gaussian. It is based on (multivariate) Analysis of Variance logic. In fact, as @Stephan Kolassa correctly noted, both the SC and the CH will reward classes with large within-class homogeneity, whereas (potentially nonlinear) classes with larger within-class variation may not be assessed as good.

  • $\begingroup$ @Majte Did you have a look at the "Simplified Silhouette" mentioned on the Wikipedia page? $\endgroup$ Feb 7 at 17:18
  • $\begingroup$ +1. Depending on the shape of the clusters, it might make sense to modify the silhouette a bit, e.g., by calculating it not over all points, but only over the 10% closest ones, otherwise the results may look strange if the data are nonlinearly separable as here. $\endgroup$ Feb 8 at 9:00
  • $\begingroup$ @Majte I added another suggestion to my answer. $\endgroup$ Feb 8 at 11:26
  • $\begingroup$ @Majte What null hypothesis do you want to test there? I had in mind the null hypothesis that that labels are independent of the continuous variables. The permutation approach can be used for this. I'm not saying that bootstrap cannot be used for this, but it isn't clear to me how you'd want to do that. $\endgroup$ Feb 12 at 18:23
  • $\begingroup$ I've implemented your solution with the null of independence. You can see the results here with a number of random scatter plots and p-values of the tests: pdfhost.io/v/LTQfunZXt_calinski The stats were derived from the permutation principle, normalized by sample size and conditioned by the number of clusters (k). Overall it looks pretty good with p<0.05 suggesting enough evidence for a visual segmentation. $\endgroup$
    – Majte
    Feb 15 at 14:04

This is an interesting and large question and no answer is likely to seem complete.

You can take the question further graphically and you can take it further numerically. Existing methods do help and so I see little or no call to invent methods ad hoc.


Your first plot already includes ellipses fitted somehow and indeed the extent to which those ellipses do or do not overlap gives a graphical handle on the question.

A once fashionable and in my view unduly neglected method plots convex hulls for each group or category, or convex hulls of points not on the convex hull, and so on -- offering compromises between inclusiveness and robustness or resistance of summary. See e.g. https://www.statalist.org/forums/forum/general-stata-discussion/general/1517556-convex-hulls-on-scatter-plots for some simple examples.

A plot like your second is likely to be seem confusing to all. Different methods include plotting groups separately in a series of small multiples or (sometimes best of all) plotting each group separately but with a backdrop of all the other points. This method has been dubbed that of front-and-back plots. See e.g. https://journals.sagepub.com/doi/pdf/10.1177/1536867X211025838


The importance of the categorical variable as a extra predictor in regression or similar models is usually best assessed by declaring as it as a factor variable to your software and fitting more complicated models in which each group may have a different intercept, or a different slope, or both. The measure of whether groups differ is how far they make different predictions of the outcome variable.

  • $\begingroup$ I am not sure about Numerics. Factor/dummies has been my very first thought but it doesn't exclude these dots overlapping. You may have a complete overlap with one slope positive and the other negative and a visualization would be looking fairly random. Convex hulls and their overlaps could be a good start it seems, but then I would need to derive the statistics and p-values from it - which is not "improbable" once we plow through the maths and see what (hopefully known) distribution we ends up with. $\endgroup$
    – Majte
    Feb 7 at 15:36
  • $\begingroup$ Having said that, what about something more simple. Could Cohen's d be used to measure the difference in the means in both dimensions? E.g. if Cohen's d is averaged 0.8 for both means in x and y it could be strong indication vs a Cohen's d of, say 0.2? $\endgroup$
    – Majte
    Feb 7 at 15:36
  • $\begingroup$ At some point you need to decide what the question is precisely and indeed if it is only about whether areas overlap then it is back to you on how to quantify overlap and how to model the generating process. If you are not interested in regression as such, fine, but then the question remains wide open. $\endgroup$
    – Nick Cox
    Feb 7 at 15:55
  • $\begingroup$ No, it's not that. I have already the underlying regression and the regressions of all sub-populations that is captured using a different visualization method. Here the question is to get the segments carved out (let's say in colors as in the image) and in a way that when visualized it doesn't produce non-sense segments and it should be clear to anyone looking at it that these are good segments. But fine, I have edited the question to "well visually defined" segments. $\endgroup$
    – Majte
    Feb 7 at 16:07
  • 1
    $\begingroup$ As I stated I would like to have a statistical method or see if anyone knows one. It would allow me to go through a large amount of datasets using code and find those with good visual segmentations within their regressions. Yes, Sextus Empiricus's answer is good, but I am not that trigger-friendly to just accept an answer yet. Let the community explore this question, maybe there are other methods that we both don't know of. $\endgroup$
    – Majte
    Feb 7 at 16:23

You need two steps:

  1. Some way of modelling the distribution of the different categories
  2. Comparing the distributions of the different categories.

There are many different ways to model distributions and to compare the difference between distributions.

A classical example would be MANOVA which models the mean and covariance matrix of the different distributions (and assumes equal covariance of the different distributions) and compares the variance within the groups and between the groups as a measure of the difference between the groups.

If the covariance for the different groups differs then you could use a quadratic classification model and use some performance measure of the model in predicting the right classes as a measure for the difference between categories.

For more fancy distributions you can use more fancy classification schemes. With a nearest neighbors algorithm you could approximate some sort of divergence measure (if I search on google with keywords 'nearest neighbours compute divergence' then I get several suggestions).


I interpreted

visually well-defined segments

as "separable in some (natural) space parametrization". I assume that for you, this is not a case of visually well-defined segments:

separable with kernelization

Further, your first image seems to suggest a GMM-type geometry, which is a natural choice. Since you already know the categorical part of your data $D$ (acting here as class assignment $k_i\in\{1...K\}$), you also have the (MLE) Gaussian Mixture Model fit $\hat{M}_D$ (no EM algorithm needed). Now you could compute a goodness of fit $T_{\hat{M}_D}$ of your model $\hat{M}_D$ by summing all pairwise Kullback-Leibler divergences $\text{KL}(\mathcal{N(\mu_i,\Sigma_i)}\,||\,\mathcal{N}(\mu_j,\Sigma_j)),\,\,(1\leq i, j \leq K)$ of Gaussians (which is never $\infty$ due to infinite support).

KL divergence

Two datasets $D_1$, $D_2$ with the same number of "classes" $K$ should be comparable by $T_{\hat{M}_{D_1}}$ and $T_{\hat{M}_{D_2}}$, where higher values of this "test statistic" (I have reasons not to call it that) indicate better visual separation.

EDIT: Normalization w.r.t. $K$ could be achieved by taking the average or maximum over the KL divergences (instead of summing). If you have a lot of datasets, you could also compare these aggregations against your (empirical) distribution of $T_{\hat{M}_{D_i}}|D_i$ to arrive at an absolute threshold/measure, similar to a p-value.


Not the answer you're looking for? Browse other questions tagged or ask your own question.