comparison of Pearson's R from samples of different size

I would like to know if it is statistically correct to compare Pearson's R correlation coefficients calculated from samples of different size. For example I have a sample of 160 observations that yield R=0.5 with the experimental values. After some filtering I end up with 130 observations and I get R=0.6. Is it correct to claim that the type of filtering I apply retains observables that tend to be more correlated to the experimental values?

• What purpose do you have in mind? It would hardly be surprising if after you filter the data the correlation changes. Would it be better to compare the two independent subsets of data? May 19, 2017 at 13:35
• I train a neural network and test it on the full test set of 160 observations and get R=0.5. When I retain the observations that fall within the domain of applicability I end up with 130 and get an R=0.6. My purpose is to prove if filtering by the domain of applicability is worth it. May 20, 2017 at 13:36

You can test this claim by randomly sampling 130 observations from your dataset many times. This implements the null hypothesis that the filtering had no relationship with correlation: that it effectively was the same as a random subsample itself.

Compute the correlation coefficient of each subsample. If the distribution of these correlation coefficients suggests $0.6$ is unusually high, you will have evidence of your claim.

Here is such an experiment performed on a dataset with a correlation coefficient of $0.5$ (shown at left as a scatterplot). The histogram of $5000$ random subsamples does not include any correlations of $0.6$ or larger. If this is how your situation turns out, you would conclude the filtering was not random and tended to select a more highly correlated subset.

It is remarkably difficult to construct a bivariate dataset of $160$ values, with correlation coefficient $0.5$, for which random subsamples of size $130$ have correlation coefficients of $0.6$ or larger. Here's one:

This dataset contains a large "core" of highly correlated values and a small cloud of "satellites" that ruin the overall correlation: it's still just $0.5$. There's a sizable chance (around $7\%$ in this case) that a sample of $130$ lies almost entirely in the core and consequently has a larger correlation. In this case, observing a correlation of $0.6$ in your particular subsample would not be considered strong evidence that it favored higher correlations.

These differing examples show why it would be wise not to apply some generic test: the details of the scatterplot can have an undue influence on the result. Don't assume the data are normally distributed or anything else: use them as they are so that you get a correct result.

In case this is looking like a lot of work, note that the R code to perform the simulation is short and simple. Suppose the data are in two arrays x and y. Here's how to get the correlations:

n.sample <- 130
r.sample <- replicate(5e3, {
i <- sample.int(length(x), n.sample)
cor(x[i], y[i])
})


The p-value for this resampling test can be computed in a similarly straightforward manner as

r <- 0.6
p.value <- mean(abs(r.sample) >= abs(r))


It returns the proportion of sample correlation coefficients that reflect a stronger correlation than your observed value r.

To compare two different correlation coefficients, I would use William's Test. As you can see, its R implementation can take into account the number of samples in each case ($n$ and $n_2$). Thus, you can run the test to compare these two correlations and see if they are significantly different or not.

• The manual page states that Williams's test assesses "the difference between two dependent correlations sharing one variable." That doesn't seem to be the situation here: these seem to be two correlations sharing both variables: one dataset appears to be a subset of the first ("after filtering"). We might consider framing the null hypothesis in this case to be that the subset of 130 observations is a simple random sample of all 160. In effect, this question is asking for the sampling distribution of $r$ in a small-population setting.
– whuber
May 19, 2017 at 16:46

A simple approach to this is to reconceptualize this problem in terms of slopes, rather than correlations and as comparing the filtered data to the unfiltered data, rather than the filtered data to the full set. Viewing it this way, you could run a simple regression model testing whether the slope for the filtered data is different from the slope of the unfiltered data: $$y_i = B_0 + B_1x_i + B_2z_i + B_3x_iz_i + e_i$$ where $z$ is a dummy code for the filter. $B_3$ assesses whether the slopes are different.