# Using anova function in r to determine whether factors have same effect I have been asked to construct a linear model where yield is the response and all other variables are explanatory.

Here is how I constructed the model:

lm1=lm(yi~factor(va)+clay+I(ra-mean(ra)))


Where yi=yield, va=variety,clay= vector of soil variable where 1 if clay, 0 if other type of variable, and ra=rainfall.

So, now I'm asked to test the hypothesis that all of the varieties of wheat have the same effect on yield.

This means that our hypothesis is $$H_0 : v_A=v_B=v_C=v_D$$ and $$H_1 : v_A\neq v_B\neq v_C\neq v_D$$

where $$v_i$$ is the parameter of variety $$i$$

for example I have this data:


yield variety soil  rainfall
1   13.4    A   sandy loam  69
2   14.0    B   loam        69
3   7.6     C   clay        77
4   11.4    D   sandy loam  67
5   10.2    A   loam        68
6   13.0    B   loam        64
7   13.1    C   sandy loam  60
8   11.6    D   sandy loam  81
9   12.3    A   sandy loam  78
10  9.1     B   sandy loam  61


So, I use the anova function that yields me this result:

Analysis of Variance Table

Response: yi
Df  Sum Sq Mean Sq F value   Pr(>F)
factor(va)        3   5.607  1.8689  0.5569 0.647109
clay              1  30.875 30.8753  9.1996 0.004612 **
I(ra - mean(ra))  1   5.833  5.8326  1.7379 0.196218
Residuals        34 114.109  3.3561


So, the F-value is 0.5569 for factor(va)

So, I compute qf(.99,df1=4,df2=34) = 3.927333 which is bigger than the F-value. So we accept $$H_0$$ that all varieties have the same effect on yield. Would this be true? We say that we fail to reject the null hypothesis, we can never prove that it is true (and therefore should not "accept" it).

Your data is consistent with the idea that all the varieties are the same (the large p-value, or small F-statistic). But the data are also consistent with an infinite number of other cases where the varieties are not exactly equal in yield (but somewhat close).

This is kind of like a "Not Guilty" verdict in a court. The defendant may be innocent, or there may just not be enough evidence (that the jury can consider) to convict. Similarly you do not have sufficient information in your data to say that there are differences in yield between the varieties, that could be because they are identical, it could also be because you just don't have enough evidence (data) to show the difference(s).

Post hoc follow-up can show how much difference could be reasonable given your data. If all the differences are small enough then you might conclude that the practical difference is too small to care about, but if they could be big enough to be interesting, then you really need more data to make a decision.

• my original dataset consists 50 variables (used in the anova function in the question). So, my conclusion is true? Also, if the expected value of yield for each variety is nearly the same, does this contribute to the fact that the effect is the same? Do the means have to be equal in order to say that varieties have same effect or can they just be very similar (close to each other in their values)? – The Poor Jew May 25 at 17:02
• i have included a box plot in my question – The Poor Jew May 25 at 17:08
• @ThePoorJew, statistical significance, or lack thereof, can never prove that things are equal (unless you have infinite data, but not always then). Your data does not rule out the possibility of all varieties being equal (and confidence intervals would rule out case of them being very different). But that is not enough to say that they are equal because there are still cases where they are similar, but not equal, that are also not ruled out by your data. Practical significance is usually subjective, consult your client and/or subject matter experts, not some guy on the internet. – Greg Snow May 26 at 16:12
• thank you, I understand – The Poor Jew May 26 at 18:44

I have no problem with @GregSnow's answer (+1), However, from discussions in comments, it's not clear to me that you understand what one can and can't claim after doing a statistical analysis---especially when the result is not to reject the null hypothesis that all means are equal.

In your experiment, you have effects of soil type and amount of rain in addition to variety of wheat. So I will focus on a simple one-factor ANOVA with four levels (varieties A,B,C,D of wheat). Thus, we have the model $$Y_{ij} = \mu + \alpha_i + e_{ij},$$ where $e_{ij} \stackrel{iid}{\sim}\mathsf{Norm}(0, \sigma).$\$

From what you show, it seems realistic to suppose:

• You have about $$r=10$$ replications in each of the four groups (because DF(Tot) = 39), which means forty plots of wheat in the experiment;
• The model standard deviation $$\sigma \approx 1.8$$ (because MS(Resi) \approx 3.36); and
• You may be interested in detecting differences in yields among varieties of about 1 on your scale (from comments and looking at boxplots).

Before you begin such an experiment it is good to know what you expect to find, the sizes of differences among means that you think it would be worthwhile to detect, and the probability that you will detect such differences using the sample sizes at hand. Using a 'power and sample size' procedure, I find that the experiment I have described above has only about probability $$0.15 = 15\%$$ of detecting differences of $$1$$ unit of yield.

Moreover, in order to have a 90% chance of detecting differences of $$1$$ unit you would need about $$r=100$$ replications per group (400 plots altogether) and for the same probability of detecting differences of $$2$$ units, you would need about $$r = 25$$ replications (100 altogether). If I am right about your experiment and expectations, this was a 'set-up' for disappointment from the start.

The experiment I describe would have a reasonably good chance of detecting yield differences of 3 or 4 units. Very roughly speaking, to this experiment, yields within 3 or 4 units of one another are essentially equal. It is not productive to discuss whether the four group means are exactly equal or whether they are within a couple of units of one another.

In a well-designed experiment, realistically detectable differences are about the same as practically important differences. If you really want/need to detect differences of 1 or 2 units, a noticeably larger (more expensive) experiment would be required. And unless there is a general consensus among people who grow and sell wheat that such small differences are genuinely of practical importance, it is difficult to imagine why anyone would want to run the larger experiment.

In fields of study where experimentation is cheap (or mindlessly subsidized), some researchers may be able to keep increasing the sizes of their experiments until they finally happen to get a P-value from some test that is below 5%. This is called "P-hacking" and it can lead to 'discovery' of effects that are not of practical importance or interest.

Examples: Here are examples of ANOVAs with simulated normal data. First, an 'under-powered' experiment with $$r=10$$ replications per level and, second, an experiment with $$r=100.$$ Of course, one experiment of each kind hardly proves my assertions above. But it happens that the experiments I ran performed as I supposed. The first fails to reject and the second finds highly significant differences.

set.seed(526)  # for reproducibility
n = 10;  sg = 1.8
x1 = rnorm(n,13,sg);   x2 = rnorm(n,13,sg)
x3 = rnorm(n,12.5,sg); x4 = rnorm(n,12,sg)
x = c(x1,x2,x3,x4);  g = rep(1:4, each=n)
oneway.test(x ~ g)

One-way analysis of means
(not assuming equal variances)

data:  x and g
F = 1.0086, num df = 3.000, denom df = 19.713,
p-value = 0.4099


.

set.seed(527)
n = 100;  sg = 1.8
x1 = rnorm(n,13,sg);   x2 = rnorm(n,13,sg)
x3 = rnorm(n,12.5,sg); x4 = rnorm(n,12,sg)
x = c(x1,x2,x3,x4);  g = rep(1:4, each=n)
oneway.test(x ~ g)

One-way analysis of means
(not assuming equal variances)

data:  x and g
F = 7.2541, num df = 3.00, denom df = 219.64,
p-value = 0.000116