Does this situation necessitate multiple comparison corrections?

Question

I am looking to run multiple t-tests on a dataset but I am unsure if the types of tests I am running would necessitate a multiple comparison correction or not. Below are essentially the tests I will be running:

1. Is profit margin for big customers is less than for smaller customers

2. Same as 1 but just US customers

3. Is profit margin less for customers who bought product class A vs product class B

4. Same as 3 but just US customers

5. Is profit margin less for customers who pay cash vs credit

6. Same as 5 but just US customers

The way I interpret multiple comparisons is that it could be a factor for tests 1,3,5 since I am looking for differences in the same metric across three different categories and then the same would be true with 2,4,6. Would this necessitate a multiple comparison correction? Since I am not doing that many comparisons on the same population would it only be necessary if I was doing more comparisons?

Answer 1

An estimate of the inflation to the Type 1 error rate for multiple tests can be found by

Inflated Type 1 error rate (Family-wise Error Rate) = $1-(1-alpha)^c$

where alpha is the Type 1 error rate per test (normally .05 or .01), and c is then number of tests.

Example: With 6 tests, if alpha = .05, the approximate inflated error rate

Family-wise Error Rate = $1-(1-alpha)^c$ = $1-(1-.05)^6$ = .264,

so probability of a Type 1 error across these 6 tests is .264.

If you decided to reduce the per-comparison alpha from .05 to .01, then the family-wise error rate would be about

Family-wise Error Rate = $1-(1-alpha)^c = 1-(1-.01)^6$ = .058.

Using alpha = .01 per test would given an acceptable overall family-wise error rate (.058) in my opinion.

If you were requested to use a traditional correction approach, the Bonferroni correction for a per comparison alpha of .05 would be .05/6 = .008333 which is within rounding of the .01 error rate suggested.

If you planned to execute a large number of tests, then I would not use any form of Type 1 error rate correction. Instead, I would use an approach that corrects for the False Discovery Rate since it does not reduce statistical power like many multiple comparison approaches for large numbers of comparisons.

Read section on the Benjamini–Hochberg procedure here

http://www.biostathandbook.com/multiplecomparisons.html

I am not saying you should use the Benjamini–Hochberg procedure, since you have only 6 tests, just noting that for those with many tests, this is a good alternative.

Bryan

Answer 2

I'm not an expert on multiple comparison corrections so will leave the "proper" answer for someone else but here are some things that cross my mind:

First you need to bear in mind what you are doing when you make a multiple comparison correction. Essentially when you perform multiple tests you increase your chances of making a type 1 error which is the probability of finding a significant result by chance. Thus when you make a multiple comparison correction you are essentially trying to keep your chance of making at least 1 type 1 error at given level which usually is 5% . To me the appropriateness of a correction will depend on how much you want to avoid or reduce the risk of a type 1 error noting that the trade off of reducing your risk of a type 1 error is that you lose power (that is your probability of seeing a significant result given there is one).

If you do decide you want to keep this risk controlled and decide to make a correction you then probably need to take care with which correction method you pick. For example your test 1 and 2 sound like they will be quite correlated which would make tests like Holms procedure quite conservative. A more appropriate method might be the Hochberg's step-up procedure.

Alternatively you could present both the raw p-values and the adjusted p-values as a pseudo "sensitivity" of your results.

Answer 3

Would this necessitate a multiple comparison correction?

First, I just want to say the study of how to correct for multiple tests is really big so I would not say there is one good general answer (although a lot of good theory) to cover all cases. In your specific case, it sounds as if all of yours tests are comparing subsets of data (e.g., "Is profit margin for big customers is less than for smaller customers") coming from the same, much larger data set.

Imagine for the moment you had a very large data set of any type and that you randomly picked two subsets of data points at a time and then compared the two subsets by a t-test (or whatever other test). By chance, some percentage of those pairs of random subsets will have a significant p-value (e.g., generally around whatever your nominal significance cutoff is). Really for any data set that is large enough, you will likely find pairs of randomly selected subsets that show significant differences because of random noise or error. This is effectively what you are trying to combat against by correcting for multiple testing.

Moving back to your case and again assuming these comparisons are drawn from the same large data set, you still run into same potential problem of an increased false positive rate even though you are picking more intelligent subsets compared to random subsets. I would recommend correcting for multiple testing in this case. It is hard to know exactly how to structure this without more detail, but since they are all drawn from the same data set I would at least correct all 6 test together.

Since I am not doing that many comparisons on the same population would it only be necessary if I was doing more comparisons?

If my assumption is correct that all of this is coming from the same larger data set, I would say it matters as long as you worry about false positives. If the data for each of your 6 tests was independently generated, you may still consider correcting for multiple hypotheses, but my particular reasoning isn't that helpful here.

Now, this all matters whether you care that you may hit a false positive. I think most people would, but if you are just looking for potential positive cases and are not too worried about a incorrect inference, then you do not really need to worry about the correction.