0
$\begingroup$

After finishing a data collection season, there's a huge dataset containing n>>1 variables (columns) and a few different observations divided by group (rows; let's say cases and controls). The task is to find variables with "LARGE" differences between cases and controls, but before starting any test, there's a special feature to consider about the dataset:

Descriptive statistics show that most of the variables have very small dispersion (i.e., all observations are very similar, so standard dev. ≈ 0)..., although a few of them (approx. 1% of all variables) do show higher SD values. Then, although differences between cases and controls could -in principle- be detected across many variables, the fact that the majority of these variables do not show large statistical dispersion makes them unatractive. That is to say, small between-group differences would not be interesting, although they were statistically significant... So:

Should only variables with high dispersion be chosen to compare groups in statistical tests? That is to say: is it tricky, if one chooses only the subset of the top-10 variables with large statistical dispersion values, and then performes group comparisons (in order to avoid doing tests whose results would have no value)? How would Bonferroni work in this case? Would the multiple testing correction take into account only the number of tests performed, with the top-10 subset variables?

Any suggestions and/or references on this topic would be appreciated. Thanks in advance.

Improve this question
$\endgroup$
5
  • 5
    $\begingroup$ If all your variables are the same kind of quantities and are expressed in the same units of measurement, then comparing SDs as you do here is legitimate. But otherwise, this makes no sense. For instance, if the standard deviation of a mass is 0.001 kilograms and the SD of a length is 100 microns, who is to say which is "large" and which not? Could you therefore edit your question to clarify this point? Also mysterious is how you appear to jump from a question about choosing variables for discrimination (evidently) to one about multiple comparisons. What's the connection? $\endgroup$
    – whuber ♦
    Commented Jul 10, 2013 at 14:46
  • 1
    $\begingroup$ Why would only variables with high dispersion be used to make comparison? Unless the reason for low dispersion is some error or problem in your sampling plan, variables with low dispersion may be key. $\endgroup$
    – Peter Flom
    Commented Jul 10, 2013 at 15:32
  • $\begingroup$ In this particular case, the only results of interest are those showing large between-group differences, as the measurement error could cause small SDs. Of note, there could be several other reasons to decide that only differences above certain threshold are interesting. $\endgroup$
    – Elabore
    Commented Jul 10, 2013 at 15:43
  • $\begingroup$ I think @whuber perfectly understood this question, as he said: "then comparing SDs as you do here is legitimate". But is it also legitimate to reject a set of variables (those with the smallest SDs)? They are actually considered "non-informative", and (for sure!) they'll give no interesting result. Could someone please comment if this variable selection process would also be legitimate? $\endgroup$
    – Elabore
    Commented Jul 10, 2013 at 17:28
  • 1
    $\begingroup$ I'm not sure whether I understood the question or not. (I was only talking about comparing one SD to another.) It seems to me that a potential explanatory variable with extremely small SD could be the most valuable for classifying cases and controls; for instance, it could be the most strongly correlated with them. References include stats.stackexchange.com/questions/9871 (which may be a duplicate of this one, depending on what you're really asking) and stats.stackexchange.com/questions/9590. A concrete example appears at stats.stackexchange.com/questions/28474. $\endgroup$
    – whuber ♦
    Commented Jul 10, 2013 at 17:39

1 Answer 1

Reset to default
1
$\begingroup$

I am also puzzled by the rest of the question but to comment on the multiple comparisons issue: Bonferroni adjustment works the way you want it to work. What I mean by that is that it is based on very general results regarding probabilities and can be applied to many different situations. It provides strong control of the family-wise error rate, which means that the probability of even one false rejection in a family of tests is at most $\alpha$ for every combination of true and false null hypotheses.

Defining what a “family” of related tests is is up to you (and the reviewers, colleagues, clients, etc. you want to convince). If your criteria to choose which tests to run are based on the same data and are somehow related to the relevant effect size, there could be some concerns about “data dredging”, “double dipping” or “capitalizing on chance”. It might then make sense to adjust the error level for all potential comparisons and not only for the tests you end up performing. This is something you have to judge for yourself, there is no “technically correct” solution and nothing about the Bonferroni correction that prevents it to be used one way or the other.

Note that there are many other simultaneous inference techniques, some similar to Bonferroni but more powerful (in particular Holm's method) and some controlling a different but possibly more relevant error rate. If you reveal more about your data and your objectives, others might also be able to suggest approaches that sidestep the issue entirely.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.