Hypothesis Testing Results of Sample Subsets Suppose I collected a dataset, which includes many sub-segmentations. I ran an hypothesis test that turned out to be insignificant overall, but significant for a few sub-segments. Can I conclude from the current result that the sub-segments are significant for my hypothesis, or should I collect a new random sample from these sub-segments and run the hypothesis test again?
 A: We're dealing with a multiple testing problem here. If you conduct a large number of individual tests, some of them are bound to turn up significant results. This doesn't necessarily mean that these individual tests are insignificant, but it means you'll need a greater level of certainty for any given test if you want to claim that it's significant.
For a conservative threshold correction, called the Bonferroni correction, you need to divide your p-value threshold for significance by the total number of tests. If any individual test has a p-value less than that, then you can reject the hypothesis for that test.
So what if your p-value isn't significant after multiple testing correction? It still doesn't prove that there's no effect, but as you pointed out in the question, it does mean you'll need another test if you want to prove that there is an effect. You can collect a random sample from a single sub-segment and test again, using the uncorrected p-value threshold this time (as long as you only test a single sub-segment). However, it's essential that the new sample be from a different data source than the original sample. Otherwise, since you're just re-testing the exact same hypothesis with the exact same data, you're bound to discover that an effect exists. (Resampling from the original data isn't sufficient to account for this problem.)
(Side note: The usual warnings about interpreting p-values apply here; be cautious about attributing any practical significance to a result merely because it has a corrected p-value < 0.05.)
