I wonder if it is valid to perform a statistical test on t-values or z-scores. For instance, if one gets two groups of t-values, every t-value is generated from an individual t-test and the value itself is used to represent "effect size or standardized effect", then one wants to know whether the effect size in groupA is significantly larger than groupB, so Mann-Whitney test on these two groups of t-values is conducted. Or in another case, one just wants to test if the mean of all t-values in groupA is significantly larger than 0, so a t-test is conducted on these t-values. Are these processes valid? It sounds weird for me to do tests on test statistics, but I don't have a theoretical backup. It seems t-value is used as the standardized effect for each individual in each group because there are multiple measurements for each individual. Then t-value is a better choice than mean for the second step test between groups, because it also considers variance. However, if the question is about whether the effect is different between two groups, the input should still be the values of the variable of interest, then a mixed-effect or hierarchical model is more appropriate than a two-step test. Am I right? Thanks a lot.
4
-
$\begingroup$ What do you mean by valid? Whether it can be done or whether it is different (and in some way less good) than using the raw data? $\endgroup$– Sextus EmpiricusCommented Jan 25, 2023 at 17:32
-
$\begingroup$ How do you get two groups of t-values? What process is generating this data? For instance, are the t-values in a single group supposed to be related (follow the same distribution)? $\endgroup$– Sextus EmpiricusCommented Jan 25, 2023 at 17:33
-
$\begingroup$ Sorry for the confusion. By valid I mean statistical legitimacy. I don't have an explicate example, let's imagine there are 50 individuals in each group, and every individual has 100 measurements. Therefore, one can get 50 t-values for each group if a t-test is done for every individual. To make things similar, we could assume individuals are independent. $\endgroup$– user1299648Commented Jan 25, 2023 at 22:58
-
1$\begingroup$ What is the hypothesis that you want to test with this group of t-values? $\endgroup$– Sextus EmpiricusCommented Jan 25, 2023 at 23:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
I am Not Sure whether this answers the question, but rather than looking at the t or z values you can look at the standardized effect size d (Cohens d). Effect sizes can be meta-analyzed and within meta analysis there is an approach called meta-regression that allows you to examine whether study variables predict differences between effect size (such as a dummy variable coding group 1 or 2).
Best Stefan
-
$\begingroup$ A measure of effect size can be a great asset to an analysis, but that really seems unrelated to the question. $\endgroup$– DaveCommented Jan 25, 2023 at 21:25
-
$\begingroup$ @Dave. My thought was that the t value is just the product of d and a factor that is defined by the sample sizes. So rather than using a number of t values in group 1 and group 2, the user can use the d values in group 1 and group 2. These d values can be predicted in a meta-regression by a dummy variable coding the group. In meta-regression you can account for the dependencies between the d's (when they come from the same sample etc). But you are right, this was too much a speculation on my side and I should have written this as a comment. $\endgroup$– StefanCommented Jan 26, 2023 at 6:52