11
$\begingroup$

I am running stats on 5 IVs (5 personality traits, extroversion, agreeableness, conscientiousness, neuroticism, openness) against 3 DVs Attitude to PCT, Attitude to CBT, Attitude to PCT vs CBT. I also added in age and gender to see what other effects there are.

I am testing to see whether personality traits can predict attitudes of the DVs.

I initially used Pearson's correlation for all variables (45 tests).

The main finding was that extroversion was correlated to attitude of PCT at p=0.05. But as I was running 45 tests I did a Bonferroni correction of alpha = 0.05/45 = 0.001, therefore making this finding insignificant.

I then ran a simple linear regression on all variables, again extroversion was significant with attitude to PCT. If I do the Bonferroni correction this it comes out insignificant again.

Questions:

  1. Do I need to Bonferroni correct at Pearson's correlation?
  2. If I do, and therefore making extroversion with attitude to PCT insignificant, is there still a point in doing linear regression?
  3. If I do a linear regression, do I need to do the Bonferroni correction for this also?
  4. Do I only report corrected valued or both uncorrected and corrected values?
Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
5
$\begingroup$

It sounds to me like this is exploratory research / data analysis, not confirmatory. That is, it doesn't sound like you started with a theory that said only extroversion should be related to PCT for some reason. So I wouldn't worry too much about alpha adjustments, as I think of that as more related to CDA, nor would I think that your finding is necessarily true. Instead, I would think about it as something that might be true, and play with these ideas / possibilities in light of what I know about the topics at hand. Having seen this finding, does it ring true or are you skeptical? What would it mean for the current theories if it were true? Would it be interesting? Would it be important? Is it worth running a new (confirmatory) study to determine if it's true, bearing in mind the potential time, effort and expense that that entails? Remember that the reason for Bonferroni corrections is that we expect something to show up when have so many variables. So I think a heuristic can be 'would this study be sufficiently informative, even if the truth turns out to be no'? If you decide that it's not worth it, this relationship stays in the 'might' category and you move on, but if it is worth doing, test it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ If he really understands what exploratory data analysis is and he doesn't take the big correlations too seriously I would agree with you. But people will concede that they are just doing exploratory analysis to filter out the wekly correlated but yet get overly excited when they see something promising. That is part of human nature. i think that doing the adjustment using FDR as the criteria is a sensible way to put the excitement under control. $\endgroup$
    – Michael R. Chernick
    Jul 27, 2012 at 23:06
  • 1
    $\begingroup$ @MichaelChernick, I don't necessarily disagree w/ you. I just wanted to put out another opinion & I often like give a big-picture, semi-philosophical, what-is-this-all-about perspective. A lot of practitioners can get bogged down in details that seem arcane to them & are left w/o a grounded understanding. $\endgroup$
    – gung - Reinstate Monica
    Jul 28, 2012 at 1:11
  • 1
    $\begingroup$ There is no disagreement here and I understand your point. I just want to add to it that if we could be dispasionate and accept statistical principle and not get personally attached to our research with a vested interest in the outcome we could do exactly what you say. But it is so hard to do. Imagine working for a pharmaceutical company having spent millions on clinical research for a particular drug and have it fail. The medical director is going to ask you to loook mat 20 different subgroups and find one which works. $\endgroup$
    – Michael R. Chernick
    Jul 28, 2012 at 1:35
  • 1
    $\begingroup$ Subgroup analysis is one od the most controversial aspects of clinical research. Without multiplicity adjustment there is no way to legitimize it and doing it post hoc makes it difficult to sell to the FDA. This is just one example from my experience in recent years that makes me sensitive to suggestions of ignoring multiplicity. $\endgroup$
    – Michael R. Chernick
    Jul 28, 2012 at 1:39
4
$\begingroup$

I think Chl has pointed you to a lot of good material and references without directly answering the question. The answer I give may be a little controversial because I know some statisticians don't believe in multiplicity adjustment and many Bayesians don't believe in p-value. In fact, I once heard Don Berry say that using the Bayesian approach, particularly in adaptive designs, controlling the type I error is not a concern. He took that back later after seeing how practically important it is to the FDA to make sure that bad drugs don't get to market.

My answer is yes and no. If you do 45 tests, you certainly need to adjust for multiplicity but not to Bonferroni because it could be far too conservative. The inflation of the type I error when you data mine for correlation is clearly an issue that got attention with the cited post "look and you shall find correlation". All three links provide great information. What I think is missing is the resampling approach to p-value adjustment as developed so nicely by Westfall and Young. You can find examples in my bootstrap book or complete details in their resampling book. My recommendation would be to consider bootstrap or permutation methods for p-value adjustment and perhaps consider false discovery rate over the stringent family-wise error rate.

Link to Westfall and Young: http://www.amazon.com/Resampling-Based-Multiple-Testing-Adjustment-Probability/dp/0471557617/ref=sr_1_1?s=books&ie=UTF8&qid=1343398751&sr=1-1&keywords=peter+westfall

A recent book by Bretz et al on multiple comparisons: http://www.amazon.com/Multiple-Comparisons-Using-Frank-Bretz/dp/1584885742/ref=sr_1_2?s=books&ie=UTF8&qid=1343398796&sr=1-2&keywords=peter+westfall

My book with the material in section 8.5 and tons of bootstrap references: http://www.amazon.com/Bootstrap-Methods-Practitioners-Researchers-Probability/dp/0471756210/ref=sr_1_2?s=books&ie=UTF8&qid=1343398953&sr=1-2&keywords=michael+chernick

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1 The reproduction of Graham Martin's Munchausen's Statistical Grid at the end of Westfall & Young says it all in a very engaging way. You can read this in the Amazon "look inside" feature. (It's almost as amusing to see Amazon offer a \$7 trade-in price for this \$150 book.) $\endgroup$
    – whuber ♦
    Jul 27, 2012 at 20:39
  • $\begingroup$ @whuber I think I saw a cartoon once sort of showing the Baron pulling himself out of a lake by his bootstraps. Efron may have been wise to call it the bootstrap since many are skeptical that it can be done in statistics just like many are skeptical about the legend of the Baron! $\endgroup$
    – Michael R. Chernick
    Jul 27, 2012 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.