0
$\begingroup$

In the following plot, I have a linear regression of 30 points, representing 10 treatments with three replicates each. As you can see, the r-squared value is quite strong (0.83) and the p-value is highly significant. But from looking at the plot, we can see that there are just two very distinct clusters of points. In the context of a real-world result, I would not consider this a very strong relationship. Is there a statistical measure I can use to describe this?
enter image description here

When I envisage a strong, highly significant relationship, I imagine something like the plot below, where we have points occurring along the majority of the trend line. So I am looking for a numerical/statistical way to say that the second plot has a more even distribution of points than the first plot (note: I have hundreds of such plots).

enter image description here

$\endgroup$
4
  • $\begingroup$ Currently means of Y are 44 and 4.1 at x = 0 and x =0.1 correspondingly. You think their relationship is not strong, then what is strong relationship in your idea? $\endgroup$
    – user158565
    Nov 19, 2018 at 1:55
  • $\begingroup$ @user158565 thank you for the comment. Can you please expand on it? Do you not agree that there are clearly two groups? Obviously drawing a line between them will get a high r2, but normally when one sees a high r2 result they would envisage a more uniform distribution of points along the line, would they not? $\endgroup$
    – J.Con
    Nov 19, 2018 at 2:05
  • $\begingroup$ Do you want to write what I wold do into Answer? $\endgroup$
    – user158565
    Nov 19, 2018 at 2:08
  • $\begingroup$ @user158565 yes, I would appreciate that. $\endgroup$
    – J.Con
    Nov 19, 2018 at 2:08

2 Answers 2

1
$\begingroup$

This may or may not be what you are looking for, but I am going to interpret your primary question as wanting an answer to this:

So I am looking for a numerical/statistical way to say that the second plot has a more even distribution of points than the first plot.

One way to do this would be to look at the amount of variance explained in k-means clustering. Let's assume there's always going to be two clumps of points. I am using R to simulate some data here, but you can use any popular data science programming language to do k-means clustering.

I simulate data from either Situation 1 or Situation 2. Here's what an example data set looks like from Situation 1:

set.seed(1839)
n <- 30
x1 <- rbeta(n, .1, .1)
y1 <- x1 * 10 + rnorm(n)
plot(x1, y1)

enter image description here

And then here's what Situation 2 looks like:

set.seed(1839)
n <- 30
x2 <- rnorm(n)
y2 <- x2 * 10 + rnorm(n, 0, 3)
plot(x2, y2)

enter image description here

k-means clustering is what it sounds like: a clustering algorithm. It tries to group all your points into a specific number of groups (in this case, 2). As you can see from the plots, Situation 1 is far more "cluster-able" than Situation 2—it is a lot easier to classify the points from Situation 1 into two groups than it is for Situation 2.

So what we can do is run both through k-means clustering with $k = 2$:

set.seed(1839)
(km1 <- kmeans(cbind(x1, y1), 2))
(km2 <- kmeans(cbind(x2, y2), 2))

k-means will assign each point to a cluster to try and minimize the variance within clusters and maximize the variance between clusters. After that, we can see how much variance is explained by the cluster. Running the code above returns a bunch of summary information, including this for Situation 1:

Within cluster sum of squares by cluster:
[1] 18.74369 27.91263
 (between_SS / total_SS =  93.1 %)

And this for Situation 2:

Within cluster sum of squares by cluster:
[1] 480.4195 489.0025
 (between_SS / total_SS =  69.5 %)

This is the same, for example, as regressing y on the assignment to cluster. We can see that the $R^2$ for this is the same as the variance explained from the k-means result in Situation 1 above:

> summary(lm(y1 ~ km1$cluster))$r.squared
[1] 0.9310095

This would show you that Scenario 1 is much more clustered than Scenario 2, or in your words, Scenario 2 "has a more even distribution of points" than Scenario 1.

You mention you have perhaps hundreds of datasets. This is a function you could use to extract the variance explained from each:

myfunc <- function(data) {
  res <- kmeans(data, 2)
  return(res$betweenss / res$totss)
}

Which would give me:

> data <- data.frame(x1, y1)
> myfunc(data)
> [1] 0.9312472

In R, you could put all of your data.frames into a list (let's call it datalist for now), and then run lapply(datalist, myfunc) to get all the explained variance numbers you need for all of your hundreds of datasets.

I'm trying to be less R-centric: This could all be done in Python, as well, using sklearn.cluster.KMeans, but it doesn't return the variance stuff you want, so you'd need to then take the clusters and plug them as predictors into a linear regression and extract the $R^2$ value (like above) using sklearn.metrics.r2_score.

$\endgroup$
3
  • $\begingroup$ This is exactly what i was looking for. Thank you very much. $\endgroup$
    – J.Con
    Nov 19, 2018 at 4:28
  • $\begingroup$ Would you say that using the BIC value from Gaussian mixture models for cluster analysis eg. summary(mclust::Mclust(iris[-5],G=2),parameters=TRUE)$bic would be a similar approach? With the lower BIC (in absolute terms) denoting greater clustering? $\endgroup$
    – J.Con
    Nov 19, 2018 at 22:14
  • 1
    $\begingroup$ @J.Con I'm afraid my knowledge of mixture models is limited; but I would assume that if you apply the same structure to all of your data sets, then a smaller loss would denote that this cluster structure fits better, but I'm not 100% sure by any means. It sounds correct, but I could be wrong. I was thinking that you might find this paper helpful, as well: arxiv.org/abs/1808.08317 $\endgroup$
    – Mark White
    Nov 19, 2018 at 22:17
1
$\begingroup$

Based on the values of coviariate $X$, they are focused on the two points: 0 and 0.1. So studying the relationship between $X$ and $Y$ is equivalent to the mean difference of response variable $Y$ when $X$ are 0 and 0.1. Given $Y$ is continue variable, t-test is good choice. From graph, it is clear that variance of $Y$ is larger at $X=0$ than that at $X=0.1$, so Welch's t-test is needed. Instead of fixed on the null hypothesis that two means are equal, you can test the null hypothesis that mean difference equal to a given value.

In fact, maybe the best way is to get Welch's t-interval for mean difference. It is a confidence interval. If your ideal strong relationship point is included in the CI, it means we cannot claim that there is strong relationship; otherwise, we can say there is enough evidence to convince that strong relationship between $X$ and $Y$ exists.

$\endgroup$
2
  • $\begingroup$ Thank you for this useful response. I wonder, can I apply Welch's t-interval to both plots I have included in the question? If so, would the results show that the second plot has a more even distribution than the first? I will update my question. $\endgroup$
    – J.Con
    Nov 19, 2018 at 2:33
  • 1
    $\begingroup$ No, t-test, Welc's t-test, and Welch's t-interval can be use for situation of two groups, like the first graph. For second graph, simple linear regression should be used. ($X$ has 10 unique values). $\endgroup$
    – user158565
    Nov 19, 2018 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.