1
$\begingroup$

I'm fairly new to data analysis. I'm working with a dataset of around 40k observations and looking at the correlation between those variables and my target variable. I'm trying to understand more about correlation coefficient (calculated using Pearson's R) and linear regression.

Assuming my dependent variable and an independent variable has a correlation coefficient of -0.16 which is by all accounts a weak correlation, if I plot these two variables using a scatter plot with a regression line I can see a downward trend. However another of my variables might share a lower correlation coefficient with my target variable but show a steeper regression line. Does this not mean that the correlation coefficient should be higher?

Thanks in advance

Improve this question
$\endgroup$
4
  • $\begingroup$ See stats.stackexchange.com/questions/2125/… $\endgroup$
    – Adrian
    Nov 21, 2020 at 23:02
  • $\begingroup$ Also see stats.stackexchange.com/questions/32464/… $\endgroup$
    – Adrian
    Nov 21, 2020 at 23:04
  • $\begingroup$ In short: the regression coefficient (in a univariate regression of Y on a single variable X) depends on the correlation and on the standard deviations of X and Y. $\endgroup$
    – Adrian
    Nov 21, 2020 at 23:04
  • 1
    $\begingroup$ None of the answers so far spells out a further basic point. Any given slope depends on the ratio of the SDs as well as the correlation, which settles the main issue. A key implication for the question is that different slopes (say, $y$ given $x_i$ and $y$ given $x_j$) are only directly comparable with each other if the units of measurement of $x_i$ and $x_j$ are identical. Statistical thinking can be muddled if attention isn't given to units and dimensions of measurement. $\endgroup$
    – Nick Cox
    Nov 22, 2020 at 11:32

3 Answers 3

Reset to default
0
$\begingroup$

The slope of the regression line is not equal to the correlation. The correlation coefficient tries to describe how correlated your variables are. Let's take an example in which we have the US measurement of a product in "feet" in one variable and the same measurement in EU/SI units of meters in another variable. We plot the variables. They are perfectly correlated because they are describing the exact same thing but they have a slope of 0.3. That's because for every foot, you have 0.3 meters. Now let's do another plot with US feet but now instead of meters let's take kilometers. Now the slope is 0.0003. Does this mean that feet are less correlated to kilometers than meters? No.

The slope therefore only tells you how much your independent variable changes per change in your dependent variable.

So if you have income in Y and education level 1, 2, 3 and 4 in X (a categorical variable) and let's say that the slope (also called coefficient) is 1000. That means that you gain an extra 1000 dollars in income per education level. For every change in X (from education 2 to 3 for example) income would increase by 1000.

This doesn't have to be true, though. Maybe education has no impact on income in your model (for example if you have insufficient data to prove the correlation) so they're not correlated at all. That's why we use tests of hypotheses (p values).

Further reading.

Disclaimer: I am not a statistician. I answer questions because I want to give back to the community that has helped me immensely and continues to do so, as not all questions go answered. Therefore please monitor my answer for edits or comments from professional statisticians

Improve this answer
$\endgroup$
1
  • $\begingroup$ The spirit is right here, but the details should be tightened up. Pedantically, 0.3, 0.0003. and so on in the example are rounded to one significant figure. Less pedantically: The slope depends on which variable goes on which axis, but the definition of slope is the rate of change in the vertical axis variable $y$, say, with the horizontal axis variable $x$, say, so the example implies metres or kilometres on vertical, feet on horizontal. The slope ... tells you how much your dependent variable changes per change in your independent variable $\endgroup$
    – Nick Cox
    Nov 22, 2020 at 11:24
0
$\begingroup$

Not necessarily. It depends on an offsetting play between amount of noise (variance) and slope steepness: steep slope with lots of noise might return lower correlation coefficient than a shallow slope with less noise. If noise is the same, the relation showing a steeper slope will also show a stronger correlation.

Long answer:

The correlation is a measure for the extent of linear dependency between two variables, not for the slope of a line. In other words, it says how much a relation between two variables is linear, not what the linear relationship actually looks like. If variables $y$ and $x$ are related by an equation of the form $y = a+bx$ with $b\ne0$, then $corr(y,x)=1$ if $b>0$ and $corr(y,x)=-1$ if $b<0$. Nothing is stated about the slope $b$, hence even the slightest slope will lead to a correlation of 1 in absolute value (on a population level)

However, with random variables, noise comes into play so that even if a relation is perfectly linear it gets 'blurred' and thus the linear dependency is not so easy to determine anymore: $y = a+bx + \varepsilon$ with $\varepsilon \sim (0,\sigma^2)$. This leads to correlation coefficients unequal to 1 in absolute value. If there are only few observations, a shallow slope $b$ might get overwhelmed by this noise, causing the linear relation to not show at all and lead to a very small correlation coefficient. A steeper slope has less chance of getting 'overwhelmed' and thus the correlation coefficient may be higher. In a world where there's lots of data points ($n \to \infty$) the correlation will approach its 'true' value, independent of the slope. But it could very well be the case that $y$ and $x$ have a shallow slope and little noise, making the linear relationship clear, while $y$ and $z$ (where $z$ is any other random variable) have a steep slope but lots of noise, making the linear relationship not so certain.

This coincidentally shows the limitations of using correlation coefficients to model relationships between multiple variables and why regression is more useful in that case. As @Paze pointed out, hypothesis testing is used to test for significance of the (regression and if needed, correlation) coefficients.

[if needed I could eventually (?) provide a graphical illustration]

Improve this answer
$\endgroup$
0
$\begingroup$

According to PennState, the slope of the simple regression line $\hat{\beta_{1}} = r \frac{s_{y}}{s_{x}}$, with $r$ being the sample correlation and $s_{y}$ and $s_{x}$ being the standard deviations of the outcome variable $y$ and the input variable $x$. You talked about an example where $y$ and $x$ could show a lower correlation but have a steeper regression line. In other words, a situation where for your target variable $y$ and for two chosen independent variables $x_{1}$ and $x_{2}$, $r_{1} < r_{2}$ but $|\hat{\beta_{11}}| > |\hat{\beta_{12}}|$ (the sample correlation between $y$ and $x_{1}$ was less than between $y$ and $x_{2}$, but the relationship between their respective slopes is flipped). Seen in this formula this is very possible (play around with holding some of the variables constant and letting others vary. You can get slopes that stay the same as correlation changes and vice versa!).

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.