0
$\begingroup$

Let's say I have results of a measurement for a large number of treatments (p-value, beta, t-stat) on samples from a large number of different populations. I want to compare how different the effect of a treatment was in different populations, of which there are 30, and then look at which treatments were most sensitive to population differences. Populations that have effects in opposite directions are a definite possibility and would be of particular interest. Unfortunately, I don't have the raw data so I can't just go back and do a test that includes population as a factor.

My thought is to get an estimate of this by taking beta divided by the p-value. By doing this, I can factor in both the absolute difference between beta coefficient values, and my certainty that the coefficient is significant. In other words, I would get the biggest difference in scores by maximizing the absolute difference between their beta-coefficient values and minimizing each of their p-values. Either increasing p-value or decreasing the absolute difference between beta-coefficients will lower the score.

The problem that I am anticipating is that increasing p-value is not considered a valid way to claim increasing likelihood of the null hypothesis being true. I'm hoping that I can get away with it by thinking of p >0.95 as "significant non-significance" as opposed to "accepting the null". Is there something wrong with how I am thinking about this? Would this give me any different information than just looking at the effect size (t-stat)?

I've looked at this thread: Why are 0.05 < p < 0.95 results called false positives? and this: https://www.reddit.com/r/askscience/comments/2yiqjq/why_can_we_not_accept_the_null_hypothesis_if_p095/

Improve this question
$\endgroup$
4
  • $\begingroup$ Is it that you want to know the standard error, but don't have access to it? Do you know how many degrees of freedom are associated w/ your betas? $\endgroup$
    – gung - Reinstate Monica
    Aug 14, 2016 at 2:59
  • $\begingroup$ I hadn't thought about factoring in standard error, but I don't have access to it. I'm pretty sure I can get the sample size though (df = n-1, right?). What I want is to identify which groups responded most differently and most similarly from one another. $\endgroup$
    – Stonecraft
    Aug 14, 2016 at 3:10
  • $\begingroup$ This sounds a bit like a meta-analysis (MA) or meta-regression (MR) problem. If I understand you correctly you have a number of regression coefficients and enough information to back-calculate their standard errors. That would be enough material for MA or MR but perhaps you could clarify your exact scientific question as I do not want to send you off on a wild-goose chase. $\endgroup$
    – mdewey
    Aug 14, 2016 at 10:56
  • $\begingroup$ I'm not doing a meta analysis, but it could be that meta-analysis methods would work. I am looking at eQTL data from a bunch of different tissues, so what I have is a really big file with a list of genetic variants and the results of statistical tests of their effect in each tissue. What I want to do is compare the effects of variants across tissues. Ultimately, I want to identify which variants are the most and least tissue-specific (as well as other stuff). $\endgroup$
    – Stonecraft
    Aug 14, 2016 at 18:06

1 Answer 1

Reset to default
0
$\begingroup$

Have you considered looking into the literature for references? I have never heard "significant non-significance" and that phrase made me chuckle. If you have a t-statistic and N, I would look up Jeff Rouder Bayes factor calculator online bayes factor calculator. This way you can compute evidence for the null. However, be sure to read a few of his papers so that you understand the approach and what you are doing.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, I just made that up. But I honestly don't understand why it's wrong to think of it that way. $\endgroup$
    – Stonecraft
    Aug 14, 2016 at 6:48
  • $\begingroup$ Are you saying that if I compute Bayes factor and then multiply it by the beta coefficient, it will do what I was trying to do? Inorder to read up on ways that people combine effect direction and effect strength with evidence strength into a single number, what terms should I search for? As you can probably tell from my question, I don't know what the word for what I'm trying to do is. $\endgroup$
    – Stonecraft
    Aug 14, 2016 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.