I am providing codes in R just an example, you can just see answers if you do not have experience with R. I just want to make some cases with examples.
correlation vs regression
Simple linear correlation and regression with one Y and one X:
The model:
y = a + betaX + error (residual)
Let's say we have only two variables:
X = c(4,5,8,6,12,15)
Y = c(3,6,9,8,6, 18)
plot(X,Y, pch = 19)
On a scatter diagram, the closer the points lie to a straight line, the stronger the linear relationship between two variables.
Let's see linear correlation.
cor(X,Y)
0.7828747
Now linear regression and pull-out R squared values.
reg1 <- lm(Y~X)
summary(reg1)$r.squared
0.6128929
Thus coefficients of the model are:
reg1$coefficients
(Intercept) X
2.2535971 0.7877698
The beta for X is 0.7877698. Thus out model will be:
Y = 2.2535971 + 0.7877698 * X
Square root of the R-squared value in regression is same as r in linear regression.
sqrt(summary(reg1)$r.squared)
[1] 0.7828747
Let's see scale effect on regression slope and correlation using the same above example and multiply X with a constant say 12.
X = c(4,5,8,6,12,15)
Y = c(3,6,9,8,6, 18)
X12 <- X*12
cor(X12,Y)
[1] 0.7828747
The correlation remain unchanged as do R-squared.
reg12 <- lm(Y~X12)
summary(reg12)$r.squared
[1] 0.6128929
reg12$coefficients
(Intercept) X12
0.53571429 0.07797619
You can see the regression coefficients changed but not R-square. Now another experiment lets add a constant to X and see what this will have effect.
X = c(4,5,8,6,12,15)
Y = c(3,6,9,8,6, 18)
X5 <- X+5
cor(X5,Y)
[1] 0.7828747
Correlation is still not changed after adding 5. Let's see how this will have effect on regression coefficients.
reg5 <- lm(Y~X5)
summary(reg5)$r.squared
[1] 0.6128929
reg5$coefficients
(Intercept) X5
-4.1428571 0.9357143
The R-square and correlation do not have scale effect but intercept and slope do. So slope is not same as correlation coefficient (unless variables are standardized with mean 0 and variance 1).
what is ANOVA and Why we do ANOVA ?
ANOVA is technique where we compare variances to make decisions. The response variable (called Y) is quantitative variable while X can quantitative or qualitative (factor with different levels). Both X and Y can be one or more in number. Usually we say ANOVA for qualitative variables, ANOVA in regression context is less discussed. May be this may be cause of your confusion. The null hypothesis in qualitative variable (factors eg. groups) is that mean of groups is not different / equal while in regression analysis we test whether slope of line is significantly different from 0.
Let's see an example where we can do both regression analysis and qualitative factor ANOVA as both X and Y are quantitative, but we can treat X as factor.
X1 <- rep(1:5, each = 5)
Y1 <- c(12,14,18,12,14, 21,22,23,24,18, 25,23,20,25,26, 29,29,28,30,25, 29,30,32,28,27)
myd <- data.frame (X1,Y1)
The data looks like follows.
X1 Y1
1 1 12
2 1 14
3 1 18
4 1 12
5 1 14
6 2 21
7 2 22
8 2 23
9 2 24
10 2 18
11 3 25
12 3 23
13 3 20
14 3 25
15 3 26
16 4 29
17 4 29
18 4 28
19 4 30
20 4 25
21 5 29
22 5 30
23 5 32
24 5 28
25 5 27
Now we do both regression and ANOVA. First regression:
reg <- lm(Y1~X1, data=myd)
anova(reg)
Analysis of Variance Table
Response: Y1
Df Sum Sq Mean Sq F value Pr(>F)
X1 1 684.50 684.50 101.4 6.703e-10 ***
Residuals 23 155.26 6.75
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
reg$coefficients
(Intercept) X1
12.26 3.70
Now conventional ANOVA (mean ANOVA for factor/qualitative variable) by converting X1 to factor.
myd$X1f <- as.factor (myd$X1)
regf <- lm(Y1~X1f, data=myd)
anova(regf)
Analysis of Variance Table
Response: Y1
Df Sum Sq Mean Sq F value Pr(>F)
X1f 4 742.16 185.54 38.02 4.424e-09 ***
Residuals 20 97.60 4.88
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
You can see changed X1f Df which is 4 instead of 1 in above case.
In contrast to ANOVA for qualitative variables, in context of quantitative variables where we do regression analysis - Analysis of Variance (ANOVA) consists of calculations that provide information about levels of variability within a regression model and form a basis for tests of significance.
Basically ANOVA tests the null hypothesis beta = 0 (with alternative hypothesis beta is not equal to 0). Here we do F test which ratio of variability explained by the model vs error (residual variance). Model variance comes from the amount explained by the line you fit while residual comes from the value that is not explained by the model. A significant F means that beta value is not equal to zero, means that there is significant relationship between two variables.
> anova(reg1)
Analysis of Variance Table
Response: Y
Df Sum Sq Mean Sq F value Pr(>F)
X 1 81.719 81.719 6.3331 0.0656 .
Residuals 4 51.614 12.904
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Here we can see high correlation or R-squared but still not significant result. Sometime you may get a result where low correlation still significant correlation. The reason of non significant relation in this case is that we do not have enough data (n = 6, residual df = 4), so the F should be looked at F distribution with numerator 1 df vs 4 denomerator df. So this case we could not rule out slope is not equal to 0.
Let's see another example:
X = c(4,5,8,6,2, 5,6,4,2,3, 8,2,5,6,3, 8,9,3,5,10)
Y = c(3,6,9,8,6, 8,6,8,10,5, 3,3,2,4,3, 11,12,4,2,14)
reg3 <- lm(Y~X)
anova(reg3)
Analysis of Variance Table
Response: Y
Df Sum Sq Mean Sq F value Pr(>F)
X 1 69.009 69.009 7.414 0.01396 *
Residuals 18 167.541 9.308
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
R-square value for this new data:
summary(reg3)$r.squared
[1] 0.2917296
cor(X,Y)
[1] 0.54012
Although the correlation is lower than previous case we got a significant slope. More data increases df and provides enough information so that we can rule out null hypothesis that slope is not equal to zero.
Lets take another example where there is negate correlation:
X1 = c(4,5,8,6,12,15)
Y1 = c(18,16,2,4,2, 8)
# correlation
cor(X1,Y1)
-0.5266847
# r-square using regression
reg2 <- lm(Y1~X1)
summary(reg2)$r.squared
0.2773967
sqrt(summary(reg2)$r.squared)
[1] 0.5266847
As values were squared square root will not provide information about positive or negative relationship here. But the magnitude is the same.
Multiple regression case:
Multiple linear regression attempts to model the relationship between two or more explanatory variables and a response variable by fitting a linear equation to observed data. The above discussion can be extended to multiple regression case. In this case we have multiple beta in the term:
y = a + beta1X1 + beta2X2 + beta2X3 + ................+ betapXp + error
Example:
X1 = c(4,5,8,6,2, 5,6,4,2,3, 8,2,5,6,3, 8,9,3,5,10)
X2 = c(14,15,8,16,2, 15,3,2,4,7, 9,12,5,6,3, 12,19,13,15,20)
Y = c(3,6,9,8,6, 8,6,8,10,5, 3,3,2,4,3, 11,12,4,2,14)
reg4 <- lm(Y~X1+X2)
Let's see the coefficients of the model:
reg4$coefficients
(Intercept) X1 X2
2.04055116 0.72169350 0.05566427
Thus your multiple linear regression model would be:
Y = 2.04055116 + 0.72169350 * X1 + 0.05566427* X2
Now lets test if the beta for X1 and X2 are greater than 0.
anova(reg4)
Analysis of Variance Table
Response: Y
Df Sum Sq Mean Sq F value Pr(>F)
X1 1 69.009 69.009 7.0655 0.01656 *
X2 1 1.504 1.504 0.1540 0.69965
Residuals 17 166.038 9.767
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Here we say that the slope of X1 is greater than 0 while we could not rule that the slope of X2 being greater than 0.
Please note that slope is not correlation between X1 and Y or X2 and Y.
> cor(Y, X1)
[1] 0.54012
> cor(Y,X2)
[1] 0.3361571
In multiple variate situation (where the variable are greater than two Partial correlation comes into the play. Partial correlation is the correlation of two variables while controlling for a third or more other variables.
source("http://www.yilab.gatech.edu/pcor.R")
pcor.test(X1, Y,X2)
estimate p.value statistic n gn Method Use
1 0.4567979 0.03424027 2.117231 20 1 Pearson Var-Cov matrix
pcor.test(X2, Y,X1)
estimate p.value statistic n gn Method Use
1 0.09473812 0.6947774 0.3923801 20 1 Pearson Var-Cov matrix
