0
$\begingroup$

I am looking to run multiple t-tests on a dataset but I am unsure if the types of tests I am running would necessitate a multiple comparison correction or not. Below are essentially the tests I will be running:

  1. Is profit margin for big customers is less than for smaller customers

  2. Same as 1 but just US customers

  3. Is profit margin less for customers who bought product class A vs product class B

  4. Same as 3 but just US customers

  5. Is profit margin less for customers who pay cash vs credit

  6. Same as 5 but just US customers

The way I interpret multiple comparisons is that it could be a factor for tests 1,3,5 since I am looking for differences in the same metric across three different categories and then the same would be true with 2,4,6. Would this necessitate a multiple comparison correction? Since I am not doing that many comparisons on the same population would it only be necessary if I was doing more comparisons?

$\endgroup$
1
$\begingroup$

An estimate of the inflation to the Type 1 error rate for multiple tests can be found by

Inflated Type 1 error rate (Family-wise Error Rate) = $1-(1-alpha)^c$

where alpha is the Type 1 error rate per test (normally .05 or .01), and c is then number of tests.

Example: With 6 tests, if alpha = .05, the approximate inflated error rate

Family-wise Error Rate = $1-(1-alpha)^c$ = $1-(1-.05)^6$ = .264,

so probability of a Type 1 error across these 6 tests is .264.

If you decided to reduce the per-comparison alpha from .05 to .01, then the family-wise error rate would be about

Family-wise Error Rate = $1-(1-alpha)^c = 1-(1-.01)^6$ = .058.

Using alpha = .01 per test would given an acceptable overall family-wise error rate (.058) in my opinion.

If you were requested to use a traditional correction approach, the Bonferroni correction for a per comparison alpha of .05 would be .05/6 = .008333 which is within rounding of the .01 error rate suggested.

If you planned to execute a large number of tests, then I would not use any form of Type 1 error rate correction. Instead, I would use an approach that corrects for the False Discovery Rate since it does not reduce statistical power like many multiple comparison approaches for large numbers of comparisons.

Read section on the Benjamini–Hochberg procedure here

http://www.biostathandbook.com/multiplecomparisons.html

I am not saying you should use the Benjamini–Hochberg procedure, since you have only 6 tests, just noting that for those with many tests, this is a good alternative.

Bryan

$\endgroup$
  • $\begingroup$ Thanks this is really helpful, the link is fantastic as well. In the tests I ran I expected H0 to be rejected and for all of my tests the p-value was incredibly small, well below .000001 even so I assume for this instance I can feel comfortable with my results. In the future though, with p-values closer to the significance cutoff I would definitely want to use some sort of correction like the Benjamini-Hochberg procedure. Would you say I am thinking about this properly? $\endgroup$ – user1723699 Mar 9 '18 at 20:39
  • 1
    $\begingroup$ Yes, except BH procedure is not viewed as a Type 1 error correction approach - it is different, it controls for false discovery. See link for some description of the difference. en.wikipedia.org/wiki/False_discovery_rate $\endgroup$ – Bryan Mar 9 '18 at 23:34
1
$\begingroup$

I'm not an expert on multiple comparison corrections so will leave the "proper" answer for someone else but here are some things that cross my mind:

First you need to bear in mind what you are doing when you make a multiple comparison correction. Essentially when you perform multiple tests you increase your chances of making a type 1 error which is the probability of finding a significant result by chance. Thus when you make a multiple comparison correction you are essentially trying to keep your chance of making at least 1 type 1 error at given level which usually is 5% . To me the appropriateness of a correction will depend on how much you want to avoid or reduce the risk of a type 1 error noting that the trade off of reducing your risk of a type 1 error is that you lose power (that is your probability of seeing a significant result given there is one).

If you do decide you want to keep this risk controlled and decide to make a correction you then probably need to take care with which correction method you pick. For example your test 1 and 2 sound like they will be quite correlated which would make tests like Holms procedure quite conservative. A more appropriate method might be the Hochberg's step-up procedure.

Alternatively you could present both the raw p-values and the adjusted p-values as a pseudo "sensitivity" of your results.

$\endgroup$
  • $\begingroup$ What do you mean multiple tests? $\endgroup$ – Deep North Mar 9 '18 at 18:32
  • $\begingroup$ Hi @DeepNorth, By multiple tests I refer to the 6 t-tests that were specified in the question $\endgroup$ – gowerc Mar 9 '18 at 18:41
  • $\begingroup$ Thanks for your input, even though it might not be a "proper" answer it helped me put everything in an easier to understand context. $\endgroup$ – user1723699 Mar 9 '18 at 20:45
1
$\begingroup$

Would this necessitate a multiple comparison correction?

First, I just want to say the study of how to correct for multiple tests is really big so I would not say there is one good general answer (although a lot of good theory) to cover all cases. In your specific case, it sounds as if all of yours tests are comparing subsets of data (e.g., "Is profit margin for big customers is less than for smaller customers") coming from the same, much larger data set.

Imagine for the moment you had a very large data set of any type and that you randomly picked two subsets of data points at a time and then compared the two subsets by a t-test (or whatever other test). By chance, some percentage of those pairs of random subsets will have a significant p-value (e.g., generally around whatever your nominal significance cutoff is). Really for any data set that is large enough, you will likely find pairs of randomly selected subsets that show significant differences because of random noise or error. This is effectively what you are trying to combat against by correcting for multiple testing.

Moving back to your case and again assuming these comparisons are drawn from the same large data set, you still run into same potential problem of an increased false positive rate even though you are picking more intelligent subsets compared to random subsets. I would recommend correcting for multiple testing in this case. It is hard to know exactly how to structure this without more detail, but since they are all drawn from the same data set I would at least correct all 6 test together.

Since I am not doing that many comparisons on the same population would it only be necessary if I was doing more comparisons?

If my assumption is correct that all of this is coming from the same larger data set, I would say it matters as long as you worry about false positives. If the data for each of your 6 tests was independently generated, you may still consider correcting for multiple hypotheses, but my particular reasoning isn't that helpful here.

Now, this all matters whether you care that you may hit a false positive. I think most people would, but if you are just looking for potential positive cases and are not too worried about a incorrect inference, then you do not really need to worry about the correction.

$\endgroup$
  • $\begingroup$ Thanks, this is really helpful and you are correct that it all comes from a larger dataset. With how you explained it I can see why this could increase my chances of a type 1 error. Luckily in this case the tests I performed had such incredibly low p-values even with a correction I would imagine I would still reject H0. In future cases where it is not so obvious it would certainly be more important. $\endgroup$ – user1723699 Mar 9 '18 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.