Let's say I have results of a measurement for a large number of treatments (p-value, beta, t-stat) on samples from a large number of different populations. I want to compare how different the effect of a treatment was in different populations, of which there are 30, and then look at which treatments were most sensitive to population differences. Populations that have effects in opposite directions are a definite possibility and would be of particular interest. Unfortunately, I don't have the raw data so I can't just go back and do a test that includes population as a factor.
My thought is to get an estimate of this by taking beta divided by the p-value. By doing this, I can factor in both the absolute difference between beta coefficient values, and my certainty that the coefficient is significant. In other words, I would get the biggest difference in scores by maximizing the absolute difference between their beta-coefficient values and minimizing each of their p-values. Either increasing p-value or decreasing the absolute difference between beta-coefficients will lower the score.
The problem that I am anticipating is that increasing p-value is not considered a valid way to claim increasing likelihood of the null hypothesis being true. I'm hoping that I can get away with it by thinking of p >0.95 as "significant non-significance" as opposed to "accepting the null". Is there something wrong with how I am thinking about this? Would this give me any different information than just looking at the effect size (t-stat)?