7
$\begingroup$

Suppose that we have continuous data $(X_1,Y_1),\dots,(X_n,Y_n)$. Suppose that $r_{x,y}$ is the Karl-Pearson correlation coefficient between $X_i$'s and $Y_i$'s. For what range of values of $r_{x,y}$, can we really decide that there may indeed be a linear relationship between $X_i$'s and $Y_i$' and proceed to predict $Y$ by using a linear regression?

I'm sure the topic concerning this question should be a well-studied one. I did a little search here; couldn't find relevant posts. Any answers to the above question or pointers to such a study is greatly appreciated.

$\endgroup$
6
  • 7
    $\begingroup$ It's a very bad idea to decide whether to do a linear regression or not based on the value of correlation. Correlation is different from regression in the sense that correlation treats the variables symmetrically and regression doesn't. As said in an answer, the regression model can be valid with any correlation value (only the precision of prediction and parameter estimation will change). Furthermore, if you decide whether to do a regression based on the correlation value, this will bias all estimators and tests in the regression. $\endgroup$ Jul 9 '21 at 9:39
  • 1
    $\begingroup$ stats.stackexchange.com/a/13317/919 addresses some aspects of this question (in the guise of $R^2,$ but that has a direct, simple connection with $r_{x,y}$). $\endgroup$
    – whuber
    Jul 9 '21 at 16:07
  • 5
    $\begingroup$ As you might imagine, a topic based on a fundamental statistical procedure that is over a century old has been extremely well studied. Your searches might not be turning up much because of some ambiguous language. In particular, could you clarify what you mean by "significant" correlation and what you mean by a "linear relationship"? Everybody seems to be responding to the former in the sense of "perform a null hypothesis test," whereas the latter suggests you might instead be looking for something like a goodness of fit statistic--which needs a completely different approach. $\endgroup$
    – whuber
    Jul 9 '21 at 18:19
  • 1
    $\begingroup$ So you are looking for a test for linear relationship. Here is a possible answers that a search "test linear relationship" yields: stats.stackexchange.com/a/239245. While the ANOVA suggestions only allow for comparing different model guesses, the plot "residual versus fitted" seems to be quite useful. An equivalent (I think) approach would be to try a local regression (LOESS), which makes no assumption about the functional dependency between $X$ and $Y$, and see whether it approximately yields a straight line. $\endgroup$
    – cdalitz
    Jul 10 '21 at 6:24
  • 1
    $\begingroup$ Try Anscombe's quartet where all four have high correlation of $0.816$ and consider whether linear regression is appropriate in each case $\endgroup$
    – Henry
    Jul 10 '21 at 8:00
10
$\begingroup$

For what range of values of rx,y, can we [...] proceed to predict Y by using a linear regression?

If the relationship is indeed linear, any value of correlation can work; linear regression behaves as it should across the entire range of correlations, including 0. You don't even need to examine the correlation beforehand (it seems to serve no purpose not already covered by the usual regression calculations).

However, that's a big if. You can get any correlation (except exactly 1 or -1) and not have linearity; a large (magnitude of) correlation doesn't necessarily imply the relationship is actually linear (nor does a small one imply that it isn't); correlation is not of itself a useful way to decide on the suitability of a linear regression model.

In the case of multiple regression, examining bivariate correlations is even more problematic, since the marginal bivariate correlations may be quite different from what you get in a multiple regression model. (See the Wikipedia articles on Simpson's paradox and omitted variable bias, for example.)

However, if you're interested in whether the regression is doing something useful in terms of prediction, we'd need to pin down precisely what is intended by "useful". In some cases that might be attributable to correlation values.

On the other hand, if you're instead asking "how do we perform a hypothesis test of a Pearson correlation?" you should probably edit the question to make that explicit. Under suitable assumptions you get a "standard" test readily available in packages - or fairly easily carried out by hand. [However, you're not limited to those specific assumptions, other tests of a Pearson correlation - including nonparametric tests - are possible.]

$\endgroup$
6
  • $\begingroup$ Thank you for the nice answer. //You can get any correlation (except exactly 1 or -1) and not have linearity; a large (magnitude of) correlation doesn't necessarily imply the relationship is actually linear (nor does a small one imply that it isn't)// Are you saying this as we are finding $r_{x,y}$ from only a finite sample? Hope it is clear that if the population (Karl-Pearson) correlation coefficient $\rho_{x,y} =0$, then we can certainly conclude there's linear relationship between $X$ and $Y$. $\endgroup$
    – Ashok
    Jul 10 '21 at 5:47
  • $\begingroup$ However, for finite samples, from hypothesis test, if it is clear that there's significant correlation, can't we conclude there's linear relationship? $\endgroup$
    – Ashok
    Jul 10 '21 at 5:48
  • 1
    $\begingroup$ We compute sample coefficients from finite samples, yes; per your original question, n is finite. You can have a significant correlation with a nonlinear relationship. e.g. consider the perfect population relationship $Y = X^2$. Now consider a sample set of $x$'s that are drawn uniformly on $(-0.5,1.5)$. When $X$ is uniform on that interval, the population correlation is about 0.89 and with say n=20 you almost always reject the null, but the relationship is still clearly nonlinear. $\endgroup$
    – Glen_b
    Jul 10 '21 at 6:07
  • 1
    $\begingroup$ As just demonstrated, even knowing the population correlation coefficient is no help on its own. In my example the population correlation was about 0.89, but the relationship is quadratic, not linear. In practice you're almost never analyzing data in a vacuum (it's not like nobody ever saw these variables before, almost always), and likely have (or can get) a good idea about the reasonableness of a linearity assumption outside your specific data. Failing that, you can always split a sample and use one part to examine relationships before choosing a model to fit to the rest. ... $\endgroup$
    – Glen_b
    Jul 10 '21 at 6:23
  • 1
    $\begingroup$ ... Alternatively, one can examine regression diagnostics after the fact to assess the suitability of the assumptions (though then reusing the same data to fit a new model and test something would have the problem of testing hypotheses suggested by the data, which screws up the properties of your tests). Best to keep model choice and tests completely separate. If in my example I sampled 35 points instead of 20 and plotted 15 (chosen at random), that would have been enough to know that a linear model would be insufficient $\endgroup$
    – Glen_b
    Jul 10 '21 at 6:25
6
$\begingroup$

There is a difference between a well-evidenced effect and a strong effect. For example, there is good evidence that eating bacon causes cancer, but the risk is low; and there is weak evidence that smoking marijuana leaf causes cancer, but the risk is probably high. (The reason for the gap is that the bacon eaters are subject to more medical surveillance than ganja smokers.)

So a useful statistical test of whether the correlation is well evidenced is not based on the correlation coefficient, but on the sample size.

Another feature of the situation that matters is how much of the variation is explained by the correlation: this is the R-squared statistic, coefficient of determination.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. The cited example is really interesting. //Another feature of the situation that matters is how much of the variation is explained by the correlation: this is the R-squared statistic, coefficient of determination.// Do you mean to say that this would help in ascertaining whether a linear relationship exists? $\endgroup$
    – Ashok
    Jul 10 '21 at 7:02
  • $\begingroup$ "Do you mean to say that this would help in ascertaining whether a linear relationship exists": The coefficient of determination is a measure of how much of the variation is accounted for by the linear relationship. It is high if the linear regression is highly predictive. If it is low, that could be because there is a lot of "noise" (other factors influencing the outcome) or because the relationship is not truely linear. $\endgroup$ Jul 12 '21 at 8:03
6
$\begingroup$

Often the term "significance" is used in the meaning "$\rho$ is statistically significantly different from zero". This is, however, not what most users of $\rho$ are interested in, because the null hypothesis that $\rho$ is exactly zero is almost certainly false. Hence even the tiniest deviation from zero becomes "significant" for a sample size that is large enough.

It is generally of more interest whether a correlation is strong. What is considered a "strong" correlation depends on the field, but here is a rule of thumb taken from an introductory textbook (here is an online reference for the same rule): \begin{eqnarray*} |\rho|\leq 0.3: & & \mbox{weak correlation}\\ 0.3 < |\rho|\leq 0.7: & & \mbox{moderate correlation}\\ |\rho|> 0.7: & & \mbox{strong correlation}\\ \end{eqnarray*} I would thus suggest, not to do a hypothesis test against $\rho=0$, but to report a confidence interval for $\rho$. You can find the formulas, e.g., here, and most statistical packages provide functions that compute it for you, for example cor.test in R. Then you can see how far this interval overlaps with the "weak" range.

$\endgroup$
5
  • 4
    $\begingroup$ I'm not sure where these rules of thumb come from or how seriously one should take them. A $r \geq .30$ would be above the 75th percentile of correlations typically observed in social and personality psychology, for example (Gignac & Szodorai, 2016). Rules of thumb will depend upon the field and application. $\endgroup$
    – awhug
    Jul 9 '21 at 13:45
  • $\begingroup$ These rules are hugely field dependent (a source for your "rules" would help understand which field you are considering with this answer). Further, your point about making a "non-existent" correlation significant by adding more data is wrong. The only non-existent correlation is \rho=0, and if it really is zero you are no more likely to get a significant result with a million observations than with five. $\endgroup$
    – JDL
    Jul 9 '21 at 16:04
  • $\begingroup$ @jdl These rules were taken from an introductory text book on statistics, but it may indeed depend on th field. Here is an online source giving the same rules: methods.sagepub.com/base/download/DatasetStudentGuide/…. The remark WRT making something "significant" by collecting more data rfers to the general problem of statistical signifcance that the null hypothesis is never true (the probability is zero that $\rho$ is exactly zero), and evn the tiniest deviation from zero will become significant for a sample size that is large enough. $\endgroup$
    – cdalitz
    Jul 9 '21 at 16:33
  • $\begingroup$ That sounds more reasonable, add both of those points to the answer and I'll take my downvote away! $\endgroup$
    – JDL
    Jul 9 '21 at 16:42
  • $\begingroup$ @jdl Thanks for insisting on the clarifications, which I have just added. $\endgroup$
    – cdalitz
    Jul 9 '21 at 17:48
5
$\begingroup$

You could use the following test to check whether there is significant correlation between $X$ and $Y$. Assume that you have the observations $(x_i,y_i), i =1,\dots,n$.

The null and alternative hypothesis are given by: $$ H_0: \, \rho = 0 \quad vs. \quad H_1: \rho \neq 0 $$ The test statistic is given by: $$ T = \sqrt{n-2}\frac{\hat{\rho}}{\sqrt{1-\hat{\rho}^2}}\overset{H_0}{\sim} t_{n-2} $$ Where $\hat{\rho}$ is the sample estimate for the correlation coefficient, i.e. $$ \hat{\rho}=\frac{\frac{1}{n}\sum_{i=1}^n((x_i-\bar{x})(y_i-\bar{y}))}{\sqrt{\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2} \cdot \sqrt{\frac{1}{n}\sum_{i=1}^n(y_i-\bar{y})^2} } $$

Thus, the null is rejected if $\vert T\vert >t_{n-2;1-\frac{\alpha}{2}}$.

$\endgroup$
8
  • $\begingroup$ I do not think that $\rho=0$ is a reasonable hypothesis for no correlation, because you would not call it a correlation if $\rho=0.1$, even if the confidence interval does not include zero. $\endgroup$
    – cdalitz
    Jul 9 '21 at 8:33
  • $\begingroup$ I think that it depends on what he intends to do with the test results. Let's say he has 10 observations and wants to check whether there is significant correlation, I think that this is a reasonable approach. Also choosing other values for $\rho$ is arbitrary. Why should he test $\rho = 0.4$ and not $\rho =0.6$ ? However, computing the confidence interval might be a better choice. $\endgroup$
    – Lars
    Jul 9 '21 at 9:29
  • $\begingroup$ @cdalitz: that very much depends on the field. In quantitative finance \rho=0.1 is often considered a very strong correlation. In experimental psychology it may be considered either strong or weak depending on context; in other fields it would be considered negligible. $\endgroup$
    – JDL
    Jul 9 '21 at 16:01
  • 7
    $\begingroup$ Significance of a correlation coefficient does not answer the question "is there a linear relation." Indeed, to evaluate the significance you must assume a linear relation! $\endgroup$
    – whuber
    Jul 9 '21 at 16:05
  • 1
    $\begingroup$ Ashok: No, that's not it. Significance merely measures the sample's ability to detect a nonzero correlation coefficient. It tells you nothing about the magnitude of that coefficient; and that coefficient, in turn, tells you little about whether the variables satisfy a linear relation. $\endgroup$
    – whuber
    Jul 10 '21 at 12:34
3
$\begingroup$

All you need is to compute the degree of freedom of the system, which is the number of participants minus $2$, and then refer to the table of critical values for $r$, which can be found here.

However, this gives you nothing but the mathematical significance, you still need to have a look at the scatterplot to see if a linear relationship really is a good guess or not.

$\endgroup$
3
  • $\begingroup$ Thank you for the answer. What is the test statistic? Any pointers to the mathematical theory concerning this? $\endgroup$
    – Ashok
    Jul 12 '21 at 2:33
  • $\begingroup$ @Ashok The test statistic is r itself here, but the test is completely equivalent to the F-test of a simple regression via ordinary least squares. If you know N, then from r you can derive F and vice versa. $\endgroup$ Jul 12 '21 at 10:20
  • $\begingroup$ Thanks. Any reference for this? $\endgroup$
    – Ashok
    Jul 13 '21 at 16:02
1
$\begingroup$

Like hypothesis testing in general, all you can do is propose a null hypothesis and calculate the probability of seeing the data given that null hypothesis. There is no point at which the data "definitely" comes from correlated sources, only some line in the sand where you decide that the data is "unlikely enough" to reject the null.

If you want to know how to calculate the p-value, you need to know the correlation coefficient and degrees of freedom (which is number of data points minus two). The formula generally given is $p = \frac{r \sqrt{n-2}}{1-r^2}$. There are many online calculators that give you $p$ given $n$ and $r$.

However, this formula is for the null hypothesis that the data is coming from normal IID. Just because this null is rejected, does not mean that there isn't some other hypothesis that doesn't involve correlation between $X$ and $Y$; if there is correlation within $X$ and $Y$, rather than between them, that increases the probability of seeing large sample correlation.

$\endgroup$
1
  • $\begingroup$ Thank you for the answer. Any pointers to the derivation of the formula? Btw, I didn't get the following. //Just because this null is rejected, does not mean that there isn't some other hypothesis that doesn't involve correlation between X and Y; if there is correlation within X and Y, rather than between them, that increases the probability of seeing large sample correlation.// $\endgroup$
    – Ashok
    Jul 11 '21 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.