Comparing significance of two tests I have read a 2011 article in Nature Neuroscience (let's call it Meta-Study A) which, if correct, seems to be at odds with my understanding of hypothesis testing. Meta-Study A is a meta-study of hypothesis testing in neuroscience. In order to explain the main point of Meta-Study A, I will now present one of the publications that is examined in Meta-Study A. I will call that publication Study B.
In Study B the authors measure the variable "fraction of active neurons" in mice of 4 different categories: mutant vs control mice, and naive mice vs mice trained in a certain task (expert mice). A significance test (e.g. MannWhitneyU) is performed for naive vs expert condition. It is found that the difference is significant in the mutant case (pval < 0.05), and not significant in the control case (pval > 0.05). The authors of Study B have summarized this finding as follows: "The percentage of neurons showing cue-related activity (significantly) increased with training in the mutant mice (P < 0.05), but not in the control mice (P > 0.05)".
According to the authors of Meta-Study A, the above claim the authors of Study B made was too strong and not justified by the performed tests. Namely, the argument in Meta-Study A is that if in one case the pval=0.049 and in another pval=0.051, it is not justified to claim that the former effect is stronger than the latter.
Question 1: Do you believe that this argument is correct? Is it not allowed/misleading to say that one number is larger than another, even if the difference is as small as 0.051 > 0.049?
Now, I would like to focus on the actual point of Meta-Study A, namely demonstrating that one effect (e.g. learning-related changes in mutant mice) is significantly larger than another (e.g. task-related changes in controls). I agree with the authors of Meta-Study A that the authors of Study B did not show this. But how to test this? In Meta-Study A (Fig 1a and 1d) they suggest subtracting the variable of interest between naive and expert mice, and then testing the difference between mutants and controls. On the first glance, this seems like a reasonable approach. But I think it is actually very wrong. What if mutants have on average 100x times more active neurons than controls, regardless of the other parameter. Then the difference in mutants would always be larger simply because it is measured on another scale. Would you agree that this approach is flawed, or am I missing something? I considered normalizing as a solution, but I am not sure it is fair, or even what exactly to normalize to.
Question 2: How to correctly test if one effect is significantly larger than another? Is it possible to directly compare p-values given the number of samples in each case? Or maybe some permutation test?
P.S. I am aware of multiple testing, namely that all p-value thresholds should be Bonferroni-corrected. I omitted this part for simplicity because it is orthogonal to the main point of the question.
Edit 1: As requested, I will provide a bit more details about the experiments themselves. A typical study with mice involves 10-20 mice, which are trained over multiple days (e.g. two weeks), once per day. Each training session involves multiple trials (typically ~100 per day). Mice learn the task by trying, and thus become expert at it towards the end of the study. Thus, such studies are interested in comparing sessions where performance of a given mouse is very high (expert) to the sessions where it is at chance level (naive). In this case, the test is not paired, as mice do a different number of trials in each session, and there is no direct link between sessions on different days. However, there are other studies which involve paired tests, for example, pre vs post-reward activity over trials. It would be great to be able to understand how to perform comparison of significance of two effects in paired and unpaired cases.
 A: It seems that all four categories of mice are disjoint (Mutant vs Ordinary, Expert vs Naive, leads to ME, MN, OE, ON) so that all tests are two-sample tests. In that case,
it is difficult to know how differences $D_m = ME-MN$ and $D_o = OE - ON.$ are found.
[Your say, "[T]hey suggest subtracting the variable of interest between naive and expert mice." What is of interest and what is being subtracted?] Does an N-mouse 'graduate' to being an E-mouse during the experiment?
Ideally, we might look at a two-way ANOVA with factors Genetics (with levels M and O) and
Practice (N and E). Then one could look at G by P interaction and perhaps resolve some
of the issues you raise. However, you mention a Wilcoxon test, so I guess someone
has decided data are not normal. Some sort of permutation test might be possible,
but it is not clear what the metric would be or exactly what would be permuted.
At one point you mention mice performing 'tasks'. How many tasks altogether for each mouse. Are the tasks the same and given in the same sequence to all four
categories of mice? And is the answer the same for both Studies A and B? To the extent that there is pairing,, by task that raises the possibility of paired tests, and
thus the possibility of resolving some of the issues you raise.
