Although the title of the question appears trivial, I would like to explain that it is not that trivial in the sense that it's different from the question of applying the same statistical test in similar datasets to test against a total null hypothesis (meta-analysis, e.g. using Fisher's method for combining p-values). What I am looking for, is a method (if it exists and if the question is valid in statistical terms) that would combine p-values from two different statistical tests (e.g. a t-test and a u-test, even if one is parametric and the other is not), applied to compare the centers of two samplings from two populations. So far I have searched a lot on the web without a clear answer. The best answer I could find was based on game theory concepts by David Bickel (http://arxiv.org/pdf/1111.6174.pdf).
A very simplistic solution would be a voting scheme. Suppose that I have two vectors of observations $A=[a_1, a_2, ..., a_n]$ and $B=[b_1, b_2, ..., b_n]$ and I want to apply several t-like statistics (t-test, u-test, even 1-way ANOVA) to test the hypothesis that the centers (means, medians etc.) of the two undlerlying distributions are equal against the hypothesis that they are not, at a significance level of 0.05. Suppose that I run 5 tests. Would it be legitimate to say that there is sufficient evidence to reject the null distribution if I have a p-value < 0.05 in 3 out of 5 tests?
Would another solution be to use the law of total probability or this is completely wrong? For example, suppose that $A$ is the event that the null distribution is rejected. Then, using 3 tests, $T_1$, $T_2$, $T_3$ (meaning that $P(T_1)=P(T_2)=P(T_3)=1/3$), would a possible value for $P(A)$ be $P(A) = P(A|T_1)P(T_1) + P(A|T_2)P(T_2) + P(A|T_3)P(T_3)$, where $P(A|T_i)$ is the probability that the null distribution is rejected under the test $T_i$.
I apologize if the answer is obvious or the question too stupid