Statistical measure for linear regression with two distinct clusters of points

Question

In the following plot, I have a linear regression of 30 points, representing 10 treatments with three replicates each. As you can see, the r-squared value is quite strong (0.83) and the p-value is highly significant. But from looking at the plot, we can see that there are just two very distinct clusters of points. In the context of a real-world result, I would not consider this a very strong relationship. Is there a statistical measure I can use to describe this?

When I envisage a strong, highly significant relationship, I imagine something like the plot below, where we have points occurring along the majority of the trend line. So I am looking for a numerical/statistical way to say that the second plot has a more even distribution of points than the first plot (note: I have hundreds of such plots).

Answer 1

This may or may not be what you are looking for, but I am going to interpret your primary question as wanting an answer to this:

So I am looking for a numerical/statistical way to say that the second plot has a more even distribution of points than the first plot.

One way to do this would be to look at the amount of variance explained in k-means clustering. Let's assume there's always going to be two clumps of points. I am using R to simulate some data here, but you can use any popular data science programming language to do k-means clustering.

I simulate data from either Situation 1 or Situation 2. Here's what an example data set looks like from Situation 1:

set.seed(1839)
n <- 30
x1 <- rbeta(n, .1, .1)
y1 <- x1 * 10 + rnorm(n)
plot(x1, y1)

And then here's what Situation 2 looks like:

set.seed(1839)
n <- 30
x2 <- rnorm(n)
y2 <- x2 * 10 + rnorm(n, 0, 3)
plot(x2, y2)

k-means clustering is what it sounds like: a clustering algorithm. It tries to group all your points into a specific number of groups (in this case, 2). As you can see from the plots, Situation 1 is far more "cluster-able" than Situation 2—it is a lot easier to classify the points from Situation 1 into two groups than it is for Situation 2.

So what we can do is run both through k-means clustering with $k = 2$:

set.seed(1839)
(km1 <- kmeans(cbind(x1, y1), 2))
(km2 <- kmeans(cbind(x2, y2), 2))

k-means will assign each point to a cluster to try and minimize the variance within clusters and maximize the variance between clusters. After that, we can see how much variance is explained by the cluster. Running the code above returns a bunch of summary information, including this for Situation 1:

Within cluster sum of squares by cluster:
[1] 18.74369 27.91263
 (between_SS / total_SS =  93.1 %)

And this for Situation 2:

Within cluster sum of squares by cluster:
[1] 480.4195 489.0025
 (between_SS / total_SS =  69.5 %)

This is the same, for example, as regressing y on the assignment to cluster. We can see that the $R^2$ for this is the same as the variance explained from the k-means result in Situation 1 above:

> summary(lm(y1 ~ km1$cluster))$r.squared
[1] 0.9310095

This would show you that Scenario 1 is much more clustered than Scenario 2, or in your words, Scenario 2 "has a more even distribution of points" than Scenario 1.

You mention you have perhaps hundreds of datasets. This is a function you could use to extract the variance explained from each:

myfunc <- function(data) {
  res <- kmeans(data, 2)
  return(res$betweenss / res$totss)
}

Which would give me:

> data <- data.frame(x1, y1)
> myfunc(data)
> [1] 0.9312472

In R, you could put all of your data.frames into a list (let's call it datalist for now), and then run lapply(datalist, myfunc) to get all the explained variance numbers you need for all of your hundreds of datasets.

I'm trying to be less R-centric: This could all be done in Python, as well, using sklearn.cluster.KMeans, but it doesn't return the variance stuff you want, so you'd need to then take the clusters and plug them as predictors into a linear regression and extract the $R^2$ value (like above) using sklearn.metrics.r2_score.

Answer 2

Based on the values of coviariate $X$, they are focused on the two points: 0 and 0.1. So studying the relationship between $X$ and $Y$ is equivalent to the mean difference of response variable $Y$ when $X$ are 0 and 0.1. Given $Y$ is continue variable, t-test is good choice. From graph, it is clear that variance of $Y$ is larger at $X=0$ than that at $X=0.1$, so Welch's t-test is needed. Instead of fixed on the null hypothesis that two means are equal, you can test the null hypothesis that mean difference equal to a given value.

In fact, maybe the best way is to get Welch's t-interval for mean difference. It is a confidence interval. If your ideal strong relationship point is included in the CI, it means we cannot claim that there is strong relationship; otherwise, we can say there is enough evidence to convince that strong relationship between $X$ and $Y$ exists.