14
$\begingroup$

I'm reading a paper and the author wrote:

The effect of A,B, C on Y was studied through the use of multiple regression analysis. A,B,C were entered into the regression equation with Y as the dependent variable. The analysis of variance is presented in Table 3. The effect of B on Y was significant, with B correlating .27 with Y.

English is not my mother tongue and I got really confused here.

First, he said he would run a regression analysis, then he showed us the analysis of variance. Why?

And then he wrote about the correlation coefficient, is that not from correlation analysis? Or this word could also be used to describe regression slope?

$\endgroup$

4 Answers 4

17
$\begingroup$

First, he said he would run a regression analysis, then he showed us the analysis of variance. Why?

Analysis of variance (ANOVA) is just a technique comparing the variance explained by the model versus the variance not explained by the model. Since regression models have both the explained and unexplained component, it's natural that ANOVA can be applied to them. In many software packages, ANOVA results are routinely reported with linear regression. Regression is also a very versatile technique. In fact, both t-test and ANOVA can be expressed in regression form; they are just a special case of regression.

For example, here is a sample regression output. The outcome is miles per gallon of some cars and the independent variable is whether the car was domestic or foreign:

      Source |       SS       df       MS              Number of obs =      74
-------------+------------------------------           F(  1,    72) =   13.18
       Model |  378.153515     1  378.153515           Prob > F      =  0.0005
    Residual |  2065.30594    72  28.6848048           R-squared     =  0.1548
-------------+------------------------------           Adj R-squared =  0.1430
       Total |  2443.45946    73  33.4720474           Root MSE      =  5.3558

------------------------------------------------------------------------------
         mpg |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
   1.foreign |   4.945804   1.362162     3.63   0.001     2.230384    7.661225
       _cons |   19.82692   .7427186    26.70   0.000     18.34634    21.30751
------------------------------------------------------------------------------

You can see the ANOVA reported at top left. The overall F-statistics is 13.18, with a p-value of 0.0005, indicating the model being predictive. And here is the ANOVA output:

                       Number of obs =      74     R-squared     =  0.1548
                       Root MSE      = 5.35582     Adj R-squared =  0.1430

              Source |  Partial SS    df       MS           F     Prob > F
          -----------+----------------------------------------------------
               Model |  378.153515     1  378.153515      13.18     0.0005
                     |
             foreign |  378.153515     1  378.153515      13.18     0.0005
                     |
            Residual |  2065.30594    72  28.6848048   
          -----------+----------------------------------------------------
               Total |  2443.45946    73  33.4720474   

Notice that you can recover the same F-statistics and p-value there.


And then he wrote about the correlation coefficient, is that not from correlation analysis? Or this word could also be used to describe regression slope?

Assuming the analysis involved using only B and Y, technically I would not agree with the word choice. In most of the cases, slope and correlation coefficient cannot be used interchangeably. In one special case, these two are the same, that is when both the independent and dependent variables are standardized (aka in the unit of z-score.)

For example, let's correlate miles per gallon and the price of the car:

             |    price      mpg
-------------+------------------
       price |   1.0000
         mpg |  -0.4686   1.0000

And here is the same test, using the standardized variables, you can see the correlation coefficient remains unchanged:

             |  sdprice    sdmpg
-------------+------------------
     sdprice |   1.0000
       sdmpg |  -0.4686   1.0000

Now, here are the two regression models using the original variables:

. reg mpg price

      Source |       SS       df       MS              Number of obs =      74
-------------+------------------------------           F(  1,    72) =   20.26
       Model |  536.541807     1  536.541807           Prob > F      =  0.0000
    Residual |  1906.91765    72  26.4849674           R-squared     =  0.2196
-------------+------------------------------           Adj R-squared =  0.2087
       Total |  2443.45946    73  33.4720474           Root MSE      =  5.1464

------------------------------------------------------------------------------
         mpg |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       price |  -.0009192   .0002042    -4.50   0.000    -.0013263   -.0005121
       _cons |   26.96417   1.393952    19.34   0.000     24.18538    29.74297
------------------------------------------------------------------------------

... and here is the one with standardized variables:

. reg sdmpg sdprice

      Source |       SS       df       MS              Number of obs =      74
-------------+------------------------------           F(  1,    72) =   20.26
       Model |  16.0295482     1  16.0295482           Prob > F      =  0.0000
    Residual |  56.9704514    72  .791256269           R-squared     =  0.2196
-------------+------------------------------           Adj R-squared =  0.2087
       Total |  72.9999996    73  .999999994           Root MSE      =  .88953

------------------------------------------------------------------------------
       sdmpg |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     sdprice |  -.4685967   .1041111    -4.50   0.000    -.6761384   -.2610549
       _cons |  -7.22e-09   .1034053    -0.00   1.000    -.2061347    .2061347
------------------------------------------------------------------------------

As you can see, the slope of the original variables is -0.0009192, and the one with standardized variables is -0.4686, which is also the correlation coefficient.

So, unless the A, B, C, and Y are standardized, I would not agree with the article's "correlating." Instead, I'd just opt of a one unit increase in B is associated with the average of Y being 0.27 higher.

In more complicated situation, where more than one independent variable is involved, the phenomenon described above will no longer be true.

$\endgroup$
0
7
$\begingroup$

First, he said he would run a regression analysis, then he showed us the analysis of variance. Why?

The analysis of variance table is a summary of part of the information you can get from regression. (What you may think of as an analysis of variance is a special case of regression. In either case you can partition the sums of squares into components that can be used to test various hypotheses, and this is called an analysis of variance table.)

And then he wrote about the correlation coefficient, is that not from correlation analysis? Or this word could also be used to describe regression slope?

The correlation is not the same thing as regression slope, but the two are related. However, unless they left a word (or perhaps several words) out, the pairwise correlation of B with Y doesn't tell you directly about the significance of the slope in the multiple regression. In a simple regression, the two are directly related, and such a relationship does hold. In multiple regression partial correlations are related to slopes in the corresponding way.

$\endgroup$
4
$\begingroup$

I am providing codes in R just an example, you can just see answers if you do not have experience with R. I just want to make some cases with examples.

correlation vs regression

Simple linear correlation and regression with one Y and one X:

The model:

y = a + betaX + error (residual) 

Let's say we have only two variables:

X = c(4,5,8,6,12,15)
Y = c(3,6,9,8,6, 18)
plot(X,Y, pch = 19)

On a scatter diagram, the closer the points lie to a straight line, the stronger the linear relationship between two variables.

enter image description here

Let's see linear correlation.

cor(X,Y)
0.7828747

Now linear regression and pull-out R squared values.

    reg1 <- lm(Y~X)
   summary(reg1)$r.squared
     0.6128929

Thus coefficients of the model are:

reg1$coefficients
(Intercept)           X 
  2.2535971   0.7877698

The beta for X is 0.7877698. Thus out model will be:

  Y = 2.2535971 + 0.7877698 * X 

Square root of the R-squared value in regression is same as r in linear regression.

sqrt(summary(reg1)$r.squared)
[1] 0.7828747

Let's see scale effect on regression slope and correlation using the same above example and multiply X with a constant say 12.

    X = c(4,5,8,6,12,15)
    Y = c(3,6,9,8,6, 18)
    X12 <- X*12

    cor(X12,Y)
   [1] 0.7828747

The correlation remain unchanged as do R-squared.

    reg12 <- lm(Y~X12)
    summary(reg12)$r.squared
     [1] 0.6128929
     reg12$coefficients
(Intercept)         X12 
 0.53571429  0.07797619 

You can see the regression coefficients changed but not R-square. Now another experiment lets add a constant to X and see what this will have effect.

    X = c(4,5,8,6,12,15)
    Y = c(3,6,9,8,6, 18)
    X5 <- X+5

    cor(X5,Y)
   [1] 0.7828747

Correlation is still not changed after adding 5. Let's see how this will have effect on regression coefficients.

        reg5 <- lm(Y~X5)
        summary(reg5)$r.squared
         [1] 0.6128929
         reg5$coefficients
(Intercept)          X5 
 -4.1428571   0.9357143

The R-square and correlation do not have scale effect but intercept and slope do. So slope is not same as correlation coefficient (unless variables are standardized with mean 0 and variance 1).

what is ANOVA and Why we do ANOVA ?

ANOVA is technique where we compare variances to make decisions. The response variable (called Y) is quantitative variable while X can quantitative or qualitative (factor with different levels). Both X and Y can be one or more in number. Usually we say ANOVA for qualitative variables, ANOVA in regression context is less discussed. May be this may be cause of your confusion. The null hypothesis in qualitative variable (factors eg. groups) is that mean of groups is not different / equal while in regression analysis we test whether slope of line is significantly different from 0.

Let's see an example where we can do both regression analysis and qualitative factor ANOVA as both X and Y are quantitative, but we can treat X as factor.

    X1 <- rep(1:5, each = 5)
    Y1 <- c(12,14,18,12,14,  21,22,23,24,18,  25,23,20,25,26, 29,29,28,30,25, 29,30,32,28,27)
   myd <- data.frame (X1,Y1)

The data looks like follows.

   X1 Y1
1   1 12
2   1 14
3   1 18
4   1 12
5   1 14
6   2 21
7   2 22
8   2 23
9   2 24
10  2 18
11  3 25
12  3 23
13  3 20
14  3 25
15  3 26
16  4 29
17  4 29
18  4 28
19  4 30
20  4 25
21  5 29
22  5 30
23  5 32
24  5 28
25  5 27

Now we do both regression and ANOVA. First regression:

 reg <- lm(Y1~X1, data=myd)
 anova(reg)

Analysis of Variance Table

Response: Y1
          Df Sum Sq Mean Sq F value    Pr(>F)    
X1         1 684.50  684.50   101.4 6.703e-10 ***
Residuals 23 155.26    6.75                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

reg$coefficients             
(Intercept)          X1 
      12.26        3.70 

Now conventional ANOVA (mean ANOVA for factor/qualitative variable) by converting X1 to factor.

myd$X1f <- as.factor (myd$X1)
     regf <- lm(Y1~X1f, data=myd)
     anova(regf)
Analysis of Variance Table

Response: Y1
          Df Sum Sq Mean Sq F value    Pr(>F)    
X1f        4 742.16  185.54   38.02 4.424e-09 ***
Residuals 20  97.60    4.88                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

You can see changed X1f Df which is 4 instead of 1 in above case.

In contrast to ANOVA for qualitative variables, in context of quantitative variables where we do regression analysis - Analysis of Variance (ANOVA) consists of calculations that provide information about levels of variability within a regression model and form a basis for tests of significance.

Basically ANOVA tests the null hypothesis beta = 0 (with alternative hypothesis beta is not equal to 0). Here we do F test which ratio of variability explained by the model vs error (residual variance). Model variance comes from the amount explained by the line you fit while residual comes from the value that is not explained by the model. A significant F means that beta value is not equal to zero, means that there is significant relationship between two variables.

 > anova(reg1)
    Analysis of Variance Table

    Response: Y
              Df Sum Sq Mean Sq F value Pr(>F)  
    X          1 81.719  81.719  6.3331 0.0656 .
    Residuals  4 51.614  12.904                 
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here we can see high correlation or R-squared but still not significant result. Sometime you may get a result where low correlation still significant correlation. The reason of non significant relation in this case is that we do not have enough data (n = 6, residual df = 4), so the F should be looked at F distribution with numerator 1 df vs 4 denomerator df. So this case we could not rule out slope is not equal to 0.

Let's see another example:

 X = c(4,5,8,6,2,  5,6,4,2,3,   8,2,5,6,3,  8,9,3,5,10)
    Y = c(3,6,9,8,6,  8,6,8,10,5,  3,3,2,4,3,  11,12,4,2,14)
    reg3 <- lm(Y~X)
    anova(reg3)


     Analysis of Variance Table

    Response: Y
              Df  Sum Sq Mean Sq F value  Pr(>F)  
    X          1  69.009  69.009   7.414 0.01396 *
    Residuals 18 167.541   9.308                  
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

R-square value for this new data:

 summary(reg3)$r.squared
 [1] 0.2917296
cor(X,Y)
[1] 0.54012

Although the correlation is lower than previous case we got a significant slope. More data increases df and provides enough information so that we can rule out null hypothesis that slope is not equal to zero.

Lets take another example where there is negate correlation:

 X1 = c(4,5,8,6,12,15)
    Y1 = c(18,16,2,4,2, 8)
   # correlation 
    cor(X1,Y1)
 -0.5266847
   # r-square using regression
    reg2 <- lm(Y1~X1)
   summary(reg2)$r.squared
 0.2773967
  sqrt(summary(reg2)$r.squared)
[1] 0.5266847

As values were squared square root will not provide information about positive or negative relationship here. But the magnitude is the same.

Multiple regression case:

Multiple linear regression attempts to model the relationship between two or more explanatory variables and a response variable by fitting a linear equation to observed data. The above discussion can be extended to multiple regression case. In this case we have multiple beta in the term:

y = a + beta1X1 + beta2X2 + beta2X3 + ................+ betapXp + error 

Example: 
    X1 = c(4,5,8,6,2,  5,6,4,2,3,   8,2,5,6,3,  8,9,3,5,10)
    X2 = c(14,15,8,16,2,  15,3,2,4,7,   9,12,5,6,3,  12,19,13,15,20)
    Y = c(3,6,9,8,6,  8,6,8,10,5,  3,3,2,4,3,  11,12,4,2,14)
    reg4 <- lm(Y~X1+X2)

Let's see the coefficients of the model:

reg4$coefficients

(Intercept)          X1          X2 
 2.04055116  0.72169350  0.05566427

Thus your multiple linear regression model would be:

Y = 2.04055116 + 0.72169350 * X1 + 0.05566427* X2 

Now lets test if the beta for X1 and X2 are greater than 0.

 anova(reg4)
    Analysis of Variance Table

    Response: Y
              Df  Sum Sq Mean Sq F value  Pr(>F)  
    X1         1  69.009  69.009  7.0655 0.01656 *
    X2         1   1.504   1.504  0.1540 0.69965  
    Residuals 17 166.038   9.767                  
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here we say that the slope of X1 is greater than 0 while we could not rule that the slope of X2 being greater than 0.

Please note that slope is not correlation between X1 and Y or X2 and Y.

> cor(Y, X1)
[1] 0.54012
> cor(Y,X2)
[1] 0.3361571

In multiple variate situation (where the variable are greater than two Partial correlation comes into the play. Partial correlation is the correlation of two variables while controlling for a third or more other variables.

source("http://www.yilab.gatech.edu/pcor.R")
pcor.test(X1, Y,X2)
   estimate    p.value statistic  n gn  Method            Use
1 0.4567979 0.03424027  2.117231 20  1 Pearson Var-Cov matrix
pcor.test(X2, Y,X1)
    estimate   p.value statistic  n gn  Method            Use
1 0.09473812 0.6947774 0.3923801 20  1 Pearson Var-Cov matrix
$\endgroup$
0
1
$\begingroup$

Analysis of variance (ANOVA) and regression are actually very similar (some would say they are the same thing).

In Analysis of variance, typically you have some categories (groups) and a quantitative response variable. You calculate the amount of overall error, the amount of error within a group and the amount of error between groups.

In regression, you don't necessarily have groups anymore, but you can still partition the amount of error into an overall error, the amount of error explained by your regression model and error unexplained by your regression model. Regression models are often displayed using ANOVA tables and it's an easy way of seeing how much variation is explained by your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.