0
$\begingroup$

What's the difference between correlation and coefficient in a regression? If the author writes "there is a positive relationship between x and y" or "there is a positive correlation between x and y" is he taking about correlation or the coefficient in a regression?

$\endgroup$

3 Answers 3

1
$\begingroup$

Il your regression is $y=\alpha+\beta x+\epsilon$, then $$\hat\beta = \frac{\sigma_{xy}}{\sigma^2_x},\quad\text{while}\quad\rho_{xy}=\frac{\sigma_{xy}}{\sigma_x\sigma_y},\quad\text{so}\quad\hat\beta=\rho_{xy}\frac{\sigma_y}{\sigma_x}$$ Since $\sigma_y/\sigma_x>0$, $\hat\beta$ and $\rho_{xy}$ have the same sign: a positive coefficient implies a positive correlation, a negative coefficient implies a negative correlation.

An example in R:

> set.seed(1234)
> x <- 1:10
> error <- rnorm(10)
> y <- 3 - 2*x + error
> fit <- lm(y ~ x)
> beta_hat <- fit$coefficients[2]; beta_hat
        x 
-2.035185 
> cov(x,y) / var(x)          # == beta_hat
[1] -2.035185
> rho <- cor(x,y); rho       # same sign as beta_hat
[1] -0.9873359
> rho * sd(y) / sd(x)        # == beta_hat
[1] -2.035185
$\endgroup$
4
  • $\begingroup$ Please define your notation! $\sigma_{xy}$ is what (etc.)? $\endgroup$ Aug 21, 2020 at 11:21
  • $\begingroup$ PS I did it in my response, maybe that's enough. $\endgroup$ Aug 21, 2020 at 11:46
  • $\begingroup$ @Lewian I suppose it is a standard notation, see <en.wikipedia.org/wiki/Covariance#Definition> $\endgroup$
    – Sergio
    Aug 21, 2020 at 12:15
  • $\begingroup$ Yes it is, and I have no problem understanding it, but you can't assume that the person who asked the question knows it as well. $\endgroup$ Aug 21, 2020 at 12:48
0
$\begingroup$

If the author writes "there is a positive relationship between x and y" or "there is a positive correlation between x and y" then they would be talking about the correlation, not necessarily about the sign of the coefficient. A positive correlation means that Y increases with increasing X. I can't comment on what test the author used, but many coefficients also reflect the sign of the correlation. For example a Pearson correlation coefficient of -1 means perfect linear anti-correlation. Still, there might be some coefficients that don't behave like this, so a "positive relationship" always refers to the sign of the correlation and not necessarily to the sign of the coefficient of the specific test.

edit: For example a Chi-Square test would not tell you the sign of the correlation, just a presence of a correlation. So if the author referred to a negative correlation it would not mean that the Chi-Square coefficient is negative.

$\endgroup$
1
  • $\begingroup$ If you want to compute the sign of a correlation, you only need to compute the correlation, not a test. $\endgroup$ Aug 21, 2020 at 11:56
0
$\begingroup$

If you have two variables $x$ and $y$, correlation measures the relation between $x$ and $y$ in a way that is symmetric between $x$ and $y$, i.e., with $\rho_{xy}$ denoting the correlation between $x$ and $y$, $\rho_{xy}=\rho_{yx}$. I'm using the notation of the answer of Sergio here. A standard regression model looks like this: $$ y=\alpha+\beta x+\epsilon, $$ with $\beta$ the regression coefficient of $x$, $\alpha$ a constant intercept term, and $\epsilon$ a random error term independent of $x$. $\beta$ is related to how strongly $y$ depends on $x$. The answer of Sergio states that $\beta=\rho_{xy}\frac{\sigma_{y}}{\sigma{x}}$, $\sigma_x$ and $\sigma_y$ being the standard deviations of $x$ and $y$, which says how they are formally related. Note that this means that $\beta$ can be large if $\rho_{xy}$ is not that large in case $\sigma_{y}$ is far bigger than $\sigma_x$. The correlation $\rho_{xy}$ measures the connection between $x$ and $y$ free of the influence of $\sigma_x$ and $\sigma_y$, whereas the regression coefficient $\beta$ depends on these.

This also implies that the regression coefficent $\beta$ is almost always different from the coefficient $\gamma$ of a regression of $x$ on $y$ in a model $$ x=\delta+\gamma y+\eta, $$ i.e., regression coefficients are not symmetric in $x$ and $y$.

In fact, the idea of regression of $y$ on $x$ (first regression) is that $y$ is a function of $x$ plus some error (random deviation), and the error appears in $y$, not in $x$ (estimating the regression will minimise the squared error of the regression line comparing the predicted $y$ by the regression with the observed $y$), whereas the second regression of $x$ on $y$ implies that there is random error in $x$, not in $y$, leading to a different minimisation problem for estimation.

I don't have time to produce and embed a graph, but regression of $y$ on $x$ puts a line in a scatterplot between $x$ and $y$ that minimises the sum of squared distances of all $y$-observations to the line in y-direction, regression on $x$ on $y$ produces a line that minimises the sum of squared distances to the line in x-direction, and correlation essentially gives the slope of a line that minimises the sum of squared distances to the line of all $(x,y)$ orthogonally to the line, after having standardised the $x$ and $y$ to remove the influence of $\sigma_x$ and $\sigma_y$.

If an author writes about correlation, chances are they mean correlation ;-). If an author writes "there is a positive relationship between x and y" it doesn't matter because correlation and regression coefficient are positive in the same cases, although they won't have the same values, see Sergio's response.

PS: Note that this becomes more difficult and somewhat different in multiple regression. If there is $y$ and more than one $x$-variable, $x_1,x_2, x_3,\ldots$, say, it actually can happen that the correlation between $y$ and $x_1$ is positive whereas the regression coefficient of $x_1$ in the multiple regression for $y$ on all $x$-variables simultaneously is negative, due to potential dependence between different $x$-variables. If an author in such a situation says "there is a positive relationship between $x_1$ and $y$", chances are they mean the regression coefficient, however this is imprecise terminology and should be avoided.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.