7
$\begingroup$

Context

I need to compare two groups of students. Students of these groups did some work, which later was evaluated. Now I have the values of the accuracy of every student. Here are the results:

0.2 0.065 0.123 0.075 0.181 0.054 0.185 0.106 0.142

and

0.269 0.357 0.2 0.221 0.275 0.277 0.253 0.127 0.246

I would like to check if the first group was better in terms of accuracy than the second group. As the samples are small I thought about using t test to compare the samples.

H0 hypothesis was that the accuracy of both groups are equal

H1 hypothesis was that the accuracy of the 1st group was greater than the accuracy of the 2nd group.

I used the following R function:

t.test(precisionAdHoc$adhoc, precisionH4U$H4U, "g", 0, FALSE, TRUE, 0.95)

where precisionAdHoc$adhoc holds the data from the first group and precisionH4U$4HU holds the data from the second group.

I got the following result:

data:  precisionAdHoc$adhoc and precisionH4U$H4U 
t = -4.3687, df = 16, p-value = 0.9998
alternative hypothesis: true difference in means is greater than 0 
95 percent confidence interval:
 -0.1701334        Inf 
sample estimates:
mean of x mean of y 
0.1256667 0.2472222 

Questions

  1. Did I use the t.test function correctly? If not, what did I get wrong? and how should I correct it?
  2. If the function was correct, how should I interpret the results?
$\endgroup$
2
  • $\begingroup$ Just out of curiousity, why did you choose to do a one-sided test? (Too late to change your mind now..) $\endgroup$
    – onestop
    Sep 18 '11 at 18:17
  • $\begingroup$ as your data points into the exact opposite direction than your hypothesis, I wonder whether the labeling of the groups or the t.test is really applied correctly. Perhaps you switched the labels?? $\endgroup$
    – Henrik
    Sep 18 '11 at 22:21
7
$\begingroup$

1) Yes, it looks to me like you used the t.test function correctly.

2) Since your p-value is 0.9998 all you can say is that you don't have enough evidence to reject your null hypothesis (that the means are equal). In other words, your evidence (the scores from both group) doesn't show that the mean of the first group is greater than the mean of the second.

Note that I was careful in answering your second question not to conclude that the means are equal. You can't conclude that from the test. Only that you don't have evidence that the first is greater than the second.

$\endgroup$
3
  • 1
    $\begingroup$ It's not quite right to say your evidence doesn't show that the means are not equal: your evidence doesn't show that the mean of the first group was greater than the mean of the second group. As you chose to do a one sided test, you can say nothing whatsoever about whether the mean of the second group is greater than the mean of the first group. $\endgroup$
    – onestop
    Sep 18 '11 at 18:21
  • $\begingroup$ Oops. You're right. I forgot he was doing a one-sided test. Thanks for the correction. $\endgroup$ Sep 18 '11 at 19:15
  • $\begingroup$ @onestop but could I say anything more if I had chosen the two sided test? $\endgroup$
    – Jakub
    Sep 18 '11 at 20:17
3
$\begingroup$

The hypothesis stated is clearly one-sided and the problem is set up in a Neyman-Pearsonian manner. Those features might be deliberate, or it might be an accidental consequence of the fact that N-P is the dominant paradigm in many introductory statistics texts. If it is deliberate then the results of the analysis should stand and the null hypothesis accepted. (Move on, nothing to see here...) However, there are alternative paradigms that are worthy of consideration.

If an effect in the observed direction is interesting (or, some might insist that I say 'would have been interesting before the experiment was done') then the one-sided test was ill-advised to begin with. If these data are part of a series of characterization experiments then the use of a decision-forcing N-P approach was inappropriate. Instead, use a Fisherian approach where the two-sided p value is as an index of the evidence against the null hypothesis (pointing to fairly strong evidence against the null in this case). Another evidence-yielding approach is to determine the likelihood function for the result which can then be updated with the evidence form subsequent tests of the same null hypothesis.

To some, these suggestions will sound outlandish and bordering on illegal. However I think that it is important to point out that many of the procedures and rules of statistical analysis come from schools of thought that are little more than expressions of opinion of strong-minded statisticians.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.