15
$\begingroup$

I would like to compare two linear regression models which represent degradation rates of a mRNA over time under two different conditions. The data for each model collected independently.

Here is the dataset.

Time (hours)    log(Treatment A)     log(treatment B)
0   2.02    1.97
0   2.04    2.06
0   1.93    1.96
2   2.02    1.91
2   2.00    1.95
2   2.07    1.82
4   1.96    1.97
4   2.02    1.99
4   2.02    1.99
6   1.94    1.90
6   1.94    1.97
6   1.86    1.88
8   1.93    1.97
8   2.12    1.99
8   2.06    1.93
12  1.71    1.70
12  1.96    1.73
12  1.71    1.76
24  1.70    1.46
24  1.83    1.41
24  1.62    1.42

These are my models:

Exp1.A.lm<-lm(Exp1$Time~Exp1$(Treatment A))
Exp1.B.lm<-lm(Exp1$Time~Exp1$(Treatment B))

Call:
lm(formula = Exp1$Time ~ Exp1$(Treatment A))

Residuals:
    Min      1Q  Median      3Q     Max 
-6.8950 -1.2322  0.2862  1.2494  5.2494 

Coefficients:
                   Estimate Std. Error t value Pr(>|t|)    
(Intercept)           74.68       6.27   11.91 2.94e-10 ***
Exp1$(Treatment A)   -36.14       3.38  -10.69 1.77e-09 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 2.97 on 19 degrees of freedom
Multiple R-squared: 0.8575, Adjusted R-squared:  0.85 
F-statistic: 114.3 on 1 and 19 DF,  p-value: 1.772e-09

Call:
lm(formula = Exp1$Time ~ Exp1$(Treatment B))

Residuals:
   Min     1Q Median     3Q    Max 
-7.861 -3.278 -1.444  3.222 11.972 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)             88.281     16.114   5.478 2.76e-05 ***
Exp1$(Treatment B)  -41.668      8.343  -4.994 8.05e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 5.173 on 19 degrees of freedom
Multiple R-squared: 0.5676, Adjusted R-squared: 0.5449 
F-statistic: 24.94 on 1 and 19 DF,  p-value: 8.052e-05

To compare these two models, I used this following code. 

anova(Exp1.A.lm,Exp1.B.lm)

Analysis of Variance Table

Model 1: Exp1$Time ~ Exp1$Exp1$(Treatment A)
Model 2: Exp1$Time ~ Exp1$Exp1$(Treatment B)
  Res.Df    RSS Df Sum of Sq F Pr(>F)
1     19 167.60                      
2     19 508.48  0   -340.88

My question is why the ANOVA analysis doesn't show an F statistics and a p.val. My apologies if this is a naive question.

Based on different slopes, the rate of degradation is different in these two models, but I would like to know how statistically significant this difference is. I hope that this makes sense.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ You may notice that the ANOVA table lists the degrees of freedom associated with the analysis as 0; you have the same number of variables in both models, that is the reason that no F or p-values can be computed. $\endgroup$
    – gung - Reinstate Monica
    Dec 15, 2012 at 0:05
  • 6
    $\begingroup$ I wouldn't bother comparing these models until after checking their goodness of fit. I think you will find in the second one that neither the response nor its logarithm are linear functions of time. This calls (seriously) into question any comparison of the slope estimates. $\endgroup$
    – whuber ♦
    Apr 11, 2013 at 3:35

2 Answers 2

Reset to default
14
$\begingroup$

If you set up the data in one long column with A and B as a new column, you then can run your regression model as a GLM with a continuous time variable and a nominal "experiment" variable (A, B). The output of the ANOVA will give you the significance of the difference between the parameters. "intercept' is the common intercept and the "experiment" factor will reflect differences between the intercepts (actually overall means) between the experiments. the "Time" factor will be the common slope, and the interaction is the difference between the experiments with respect to the slope.

I have to admit I cheat (?) and run the models separately first to get the two sets of parameters and their errors and then run the combined model to acquire the differences between the treatments (in your case A and B)...

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ This is a clever approach. When you "cheat," do you check that the error variances are approximately the same in each model? And if they appear substantially different, how does that affect your recommendations? $\endgroup$
    – whuber ♦
    Apr 11, 2013 at 3:36
  • $\begingroup$ Th GLM is a good approach, and for exploring the data, fitting separate models is a good way to judge error variance between experiments. If one were really concerned, they could extend the GLM model to include group specific error variances rather than the implicit assumption of a common error variance for all experimental data. $\endgroup$
    – prince_of_pears
    Sep 22, 2018 at 12:33
  • $\begingroup$ Another thing that comes to mind is whether OP is interested in trading whether degradation rates between experiments are simply different from each other (ignoring the absolute rate), or whether these rates are also statistically (or practically) different from zero. The first amounts to a test of the hypothesis that the interaction coefficient between treatment and time is equal to zero. The second is to perform either two separate tests (or one joint hypothesis test) that each rate is different from zero. I might be more interested in testing the second before the first. $\endgroup$
    – prince_of_pears
    Sep 22, 2018 at 12:44
  • $\begingroup$ Is this method equivalent to the Chow test? $\endgroup$
    – jdcrossval
    Dec 4, 2022 at 19:24
5
$\begingroup$

The ANOVA analysis doesn't show an F statistics and a p.value since both models have the same residual degrees of freedom (i.e. 19) and if you take the difference then it would be zero! There should be at least one degree of freedom after you take the difference in order to perform the F-test.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am not sure if I understand your answer. Is there a reason that the residual degrees of freedom are equal? Any suggestion on an alternative approach to compare the slopes? $\endgroup$
    – Rooz
    Dec 10, 2012 at 10:01
  • $\begingroup$ Yes there is. In both your models you have 21 observations i.e $n=21$. In the linear regression with one explanatory random variable (i.e. either Exp1$(Treatment A) or Exp1$(Treatment B)), the degree of freedom for the regressors is one. The total degree of freedom is $df_{T}=n-1=20$. Note that $df_{T}=df_{error}+df_{regressors}$. So $df_{error}=19$ in both models, therefore you can not use the F test here. $\endgroup$
    – Stat
    Dec 10, 2012 at 21:43
  • $\begingroup$ There are many ways to compare them other than F-test. The easiest one is to use Multiple R-squared and Adjusted R-squared as you have in the summaries.The model with higher R-squared or Adjusted R-squared is better. Here the better model seems to be the one with Exp1$(Treatment A). But remember, that you should check the residuals of your model to check the adequacy of the fitted model. I personally don't recommend to rely only on R-squared criteria and you should check other assumptions in the linear model as well. Especially see if the residuals are autocorrelated or not. $\endgroup$
    – Stat
    Dec 10, 2012 at 21:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.