Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite
  1. Came across an article on the web on "evaluation of training effectiveness". The author suggests that the "t-value" obtained from a "Paired t-test" conducted using pre-test and post test scores can be used to quantify the effectiveness of training.
  2. The t-value was termed as the "Index of Learning" and the author claimed the following:
    a. An "Index of learning" (t-value) between 0 to 1.5 indicates no evidence of learning.
    b. A t-value between 1.5 to 2.0 indicates some evidence of learning.
    c. A t-value between 2 to 3 indicates strong evidence of learning.
    d. A t-value above 3 indicates very strong strong evidence of learning.
  3. In my limited understanding of the subject, the t-value, critical t-value and p-value of a t-test would depend on the significance level and degrees of freedom of the associated data. I have thus not understood the rationale behind the generalization made by the author (Para 2 (a) to 2 (c) above). I would be extremely grateful for any explanations. Thanks.
share|improve this question

2 Answers

up vote 3 down vote

The difficulty here appears to stem from terminology and convention.

First let's just define terms in terms of statistical theory. We observe a data $X_1 ... X_n$, each a random variable. We define a statistic as a function of that data, $S(X_1...X_n)$. As a function of random variables, the statistic is also a random variable. So far this constitutes the estimation part of statistics.

Certain statistic functions are better suited for detecting certain characteristics. In the case of the paired t-value, a statistic, it is designed to detect differences in the mean between 2 populations. But how do we know when $S$ is "abnormal", or "substantially different"? Here like most other things in like we go to some well-defined set of reference values, and compare our obtained statistic with the critical statistic. This part constitutes the inference part of statistics.

In this case, if we make the assumption that the data is independent, normally distributed, then we can go to the t-value table to look up the corresponding critical t-value. If the t-value is more extreme than the critical value then we can call it statistically significant, and vice versa.

This division between estimation and inference is the difference.

The paper you referred to appeared more interested in the estimation of differences in learning. It may be of interest to make the determination--the inference--of whether of not the learning was statistically significant. But that means you end up with a binary determination of "yes, significant" or "no, not significant". Letting the estimation--the t-values--be the measurement allows the user to quantitatively compare paired units.

The same thing is done for estimating, for example, bone density. The measured bone density (DEXA), a number, is converted into a t-value based on age- and ethnicity-controlled reference population. Wikipedia article. The t-value is then used to make the diagnosis of osteoporosis (t<-2.5) versus osteopenia (t between -2.5 and -1)

share|improve this answer
Dear Mr Ming, I am grateful for the trouble taken by you to post a detailed explanation. Am a little confused though. Checked up on some websites regarding Bone Density. The t-values specified in that context are in relation to standard, predefined bone density figures. Comparative t-values in terms of Std Deviations would thus make sense. In my opinion, it may not be proper to adopt a similar analogy in case of paired t-test used to evaluate pre/ post test scores simply because there are no standard/predefined figures to compare with. What do you think? Thanks anyway. – user13226 Aug 11 '12 at 18:22
Got it. I'm of the opinion that the author is trying to quantify the degree of learning (hence the nomenclature of "Index of Learning"), and not so much the weight of evidence. But I may be wrong, in which case Dr. Chernick's answers will be more relevant. My argument is that whether 1-sample t, 1-sample z, 2-sample t, or whatever statistic one comes up with, it is a standardized score that may be helpful in quantifying size of effect versus strength of evidence (which appears to be the gist here). Hope this helps. – Ming-Chih Kao Aug 12 '12 at 3:33
up vote 1 down vote

I don't think this is an issue of estimation versus inference. I think the author of the article is being a little oversimplistic. He is trying to put meaning to specific values of a t statistic as those it is a fixed measure rather than a random qusntity. If the idea is to identify what is a high value versus an intermediate or low value then you should consider the distribtuion of the t when there is no true difference. Then a large value would indicate that the difference is more likely real than a chance event. This is like doing statistical inference.

So I think ignoring the sample size and the dependence of the t statistic on degrees of freedom is as you point out a flaw in the author's argument.

share|improve this answer
Dear Mr Michael,Thanks for your response. Would like to share some observations with you and seek your comments. Carried out the t-test using excel on a fictitious data of 30 paired pretest and post test scores(Variable1-mean 42.8,var4.4,Variable2-mean 44.63,var 10.5,Pearson Correlation 0.65, t Stat 4.074, P(one tail) 0.0002, t critical(one tail) 1.69. Going by the logic of the author, this would indicate a very strong evidence of learning even though the difference in pretest and posttest scores is actually restricted to just 3 to 4 marks in the (fictitious)group of 30 students! Thanks a ton. – user13226 Aug 11 '12 at 18:42
The p-value refers to statistical significance and not practical significance. For statistical significance the magnitude of the difference does not have to be large, but rather the mean difference relative to the standard deviation of the mean difference. What you have is clear statistical significance. – Michael Chernick Aug 11 '12 at 19:28
Dear Mr Michael, with reference to your last comment, do you think the author is justified in defining an "Index of Learning" based on the t-values and cite it as "Evidence of Learning"? Please note that the author does not even talk about the p-values.He has just spoken of the t-values while defining the"Index of Learning" and relating it to evidence of learning. The data provided by me was generated using excel data analysis. The author's model suggests calculation of t-values only and brackets the same in various categories-(0 to 1.5,1.5 to 2,2 to 3 and greater than 3). Thanks a ton. – user13226 Aug 12 '12 at 3:06
I don't think the categorization is justified and if it were it is not done properly. – Michael Chernick Aug 12 '12 at 4:07
Dear Mr Michael, Thanks for all the trouble taken - prompt responses to say the least. – user13226 Aug 12 '12 at 5:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.