Statistics on results of multiple statistical tests

Question

Let's say we have multivariate data. Particular variables are measurements at certain feature points of physical specimens we are studying. The problem is traditionally a small sample size. The results are highly dependent on what feature point we decide to include into our experiment (what configuration of them).

We would like to increase robustness of our results (i.e. the results should not depend on the decision about the feature point configuration). The result is actually proving (or not) statistical significance of difference between groups of specimens (described by feature points).

Is it correct to base our result on a test made on count of positive results of many multivariate tests done on all (or many randomly chosen) configurations of measurements?

Example: Let's say we have 100 specimens split into 2 groups equal in size. Each described by 100 measurements. I decided to randomly select 20 measurements (this choice is another question). I repeat this random choice as many times as possible (all possible subsets, etc.). And I will do multivariate test for mean equality between the groups (Hotelling T2, permutation variant of it) for each subset of measurements. And I will count significant differences against the total number choices tests made. Is it possible to evaluate statistical significance of this ratio using binomial distribution?? And use it to draw conclusion?

Interpretation (numbers are made up): 1) E.g. 1500 from 5000 experiments are statistically significant (proves difference in groups). The ratio is significant, but in favor of not enough significant differences... hence difference between groups is not proved. 2) With 2500 out of 5000, ratio is not significant, we cannot say anything. Result depends on selection of feature points. 3) With 3500 out of 5000 ration is significant in favor of statistically significant differences proved by multivariate tests.

Is my description clear? Is my proposed solution valid? Is there another way to approach to this problem (multivariate data, small sample size)?

Answer 1

The problem of comparing $p$-values across experiments and studies has been a big topic in meta-analysis and one of the main conclusions is that counting significant $p$-values is always a bad idea. Instead, researchers should report and consider the effect size (e.g. Cumming, 2013). Recent approaches use hierarchical or random-effect models which I would also recommend for your case. I suggest placing the hierarchical prior on the features not on groups of features. Gelman and Hill (2006) provide an introduction to hierarchical modeling.

Cumming, G. (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge.

Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge university press.