Let me give the details of my problem setting, we have a learning algorithm that can take a collection of algebra problems and automatically generate similar ones. Of course, an important question to ask is are the problems we generated indeed similar to the original ones. We invited 20 students to answer the 10 questions for each of the problem set: original problems and generated problems, and then we use a paired t test to compare the error rate of each student across the two problem sets. The p is calculated to be 0.1 for the two mean value 10.3% (original) and 14.2 (generated), now my question is does this suffice to show the two sets of questions are similar? If not, why and what should we do? Thanks very much.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
No, it doesn't establish similarity, and indeed, it doesn't come close to answering the right question.
i) Failure to reject the null doesn't imply it's true. It simply implies your sample was too small to detect the difference.
ii) You're not actually even interested in the truth of the null you're testing. I presume you don't believe the distribution of errors is actually identical ... you must know that that would be an astronomically unlikely situation -- the new problems will have at least subtle differences from the original. You yourself use the word "similar" and the hypothesis you're testing isn't about that. With a large enough sample size, any difference in mean - no matter how trivial - might be rejected.
You may want to consider whether a confidence interval for the difference is a better tool. You can specify what you regard as the largest size of difference in error rate that is still reasonably consistent with being "similar", and see if the confidence interval contains it.
Alternatively, you might consider an equivalence test.
(I'm not 100% sure I'd use a t-test for this, since the number of errors will be fairly discrete, and there's also a potential issue with heteroskedasticity -- the variance of a difference will tend to be larger for students whose mean number of errors is larger -- but it may do well enough.)
Of course this only detects situations where the mean difference is away from zero.
It's possible that the variation is such that some people score higher and some score lower (for example, weaker students may typically score lower than they do on the original test and stronger students typically score higher than on the original test) and a paired test of means won't detect that. If you want to be able to identify that kind of change, you will want a different test.
$\begingroup$
$\endgroup$
5
No. You cannot use a t-test to prove the null hypothesis. A $p > \alpha$ only means that you do not have sufficient evidence to reject the null hypothesis.
You could try reporting a simple correlation between the two tests.
-
1$\begingroup$ Correlation is no use. If the new test precisely doubled the proportion of errors, the correlation would be 1 $\endgroup$– Glen_bCommented Oct 17, 2014 at 4:25
-
$\begingroup$ ... to clarify -- I mean if it doubled the proportion of errors for each individual. $\endgroup$– Glen_bCommented Oct 17, 2014 at 5:05
-
$\begingroup$ @Glen_b, I understand exactly what you mean but does it help if the mean is close? $\endgroup$ Commented Oct 17, 2014 at 5:16
-
1$\begingroup$ user1935724 - The correlation is completely unrelated to whether the means are close. The closeness of the means is what you seem to be concerned about. $\endgroup$– Glen_bCommented Oct 17, 2014 at 7:26
-
$\begingroup$ ... to clarify: If the means are very close, the correlation still doesn't quite do what you need (imagine $y_i-\bar{y} = 0.01 (x_i-\bar{x})$ - the correlation is 1, but the tests are still quite different). Something based on the squared pair-differences (or even absolute pair-differences) would seem to be more relevant, since they'll pick up difference in means when the means differ and also low correlation if the means and variances are identical. Perhaps something based on mean square error. $\endgroup$– Glen_bCommented Oct 17, 2014 at 7:32