- Came across an article on the web on "evaluation of training effectiveness". The author suggests that the "t-value" obtained from a "Paired t-test" conducted using pre-test and post test scores can be used to quantify the effectiveness of training.
- The t-value was termed as the "Index of Learning" and the author claimed the following:
a. An "Index of learning" (t-value) between 0 to 1.5 indicates no evidence of learning.
b. A t-value between 1.5 to 2.0 indicates some evidence of learning.
c. A t-value between 2 to 3 indicates strong evidence of learning.
d. A t-value above 3 indicates very strong strong evidence of learning.
- In my limited understanding of the subject, the t-value, critical t-value and p-value of a t-test would depend on the significance level and degrees of freedom of the associated data. I have thus not understood the rationale behind the generalization made by the author (Para 2 (a) to 2 (c) above). I would be extremely grateful for any explanations. Thanks.
The difficulty here appears to stem from terminology and convention.
First let's just define terms in terms of statistical theory. We observe a data $X_1 ... X_n$, each a random variable. We define a statistic as a function of that data, $S(X_1...X_n)$. As a function of random variables, the statistic is also a random variable. So far this constitutes the estimation part of statistics.
Certain statistic functions are better suited for detecting certain characteristics. In the case of the paired t-value, a statistic, it is designed to detect differences in the mean between 2 populations. But how do we know when $S$ is "abnormal", or "substantially different"? Here like most other things in like we go to some well-defined set of reference values, and compare our obtained statistic with the critical statistic. This part constitutes the inference part of statistics.
In this case, if we make the assumption that the data is independent, normally distributed, then we can go to the t-value table to look up the corresponding critical t-value. If the t-value is more extreme than the critical value then we can call it statistically significant, and vice versa.
This division between estimation and inference is the difference.
The paper you referred to appeared more interested in the estimation of differences in learning. It may be of interest to make the determination--the inference--of whether of not the learning was statistically significant. But that means you end up with a binary determination of "yes, significant" or "no, not significant". Letting the estimation--the t-values--be the measurement allows the user to quantitatively compare paired units.
The same thing is done for estimating, for example, bone density. The measured bone density (DEXA), a number, is converted into a t-value based on age- and ethnicity-controlled reference population. Wikipedia article. The t-value is then used to make the diagnosis of osteoporosis (t<-2.5) versus osteopenia (t between -2.5 and -1)
I don't think this is an issue of estimation versus inference. I think the author of the article is being a little oversimplistic. He is trying to put meaning to specific values of a t statistic as those it is a fixed measure rather than a random qusntity. If the idea is to identify what is a high value versus an intermediate or low value then you should consider the distribtuion of the t when there is no true difference. Then a large value would indicate that the difference is more likely real than a chance event. This is like doing statistical inference.
So I think ignoring the sample size and the dependence of the t statistic on degrees of freedom is as you point out a flaw in the author's argument.