5
$\begingroup$

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons following an omnibus ANOVA test, then the collection of post-hoc comparisons would be the "family". As the number of tests increases, the risk of Type I error also increases. Hence, family-wise error corrections (like Bonferroni) adjust the alpha criterion in order to account for this Type I error inflation. For example, if I were to perform 6 tests, then I might divide the normative 0.05 alpha by 6 to obtain an adjusted alpha criterion of 0.008, and use this adjusted alpha to determine significance.

Question: Does the family-wise error logic also apply to effect size calculations? If so, are there any common correction procedures like Bonferroni that can be used to adjust effect sizes like eta-squared or Cohen's D? If not, why not?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ The short answer is: yes, the same issue applies. If you e.g. look at the maximum of several outcome measures, it does matter how many you looked at. There are a number of corrections in specific settings (e.g. play-the-winner clinical trial designs), but I am not sure whether there are really general approaches that are as straightforward to apply as e.g. a Bonferroni correction (or perhaps more appropriately a Bonferroni-Holm). $\endgroup$
    – Björn
    Mar 4, 2016 at 16:44

4 Answers 4

Reset to default
3
$\begingroup$

Penalized maximum likelihood estimation is a good approach that leads to enhanced ability to take a point estimate out of context and have it not be overstated. For example, if one selects the group whose mean is farthest from the others, the result will be significantly biased, and penalization reduces this bias. James-Stein estimators also work, and the best of all approaches is a Bayesian hierarchical model because that is one of the few methods that allows full statistical inference to be carried out in the presence of shrinkage.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I wonder if there's a particular reason why family-wise error adjustment techniques (like Bonferonni) or the Benjamini–Hochberg procedure are still used for post-hoc multiple comparisons with ANOVAs. Aren't Bayesian hierarchical/penalized regression approaches superior? $\endgroup$
    – RobertF
    Mar 4, 2016 at 20:51
  • $\begingroup$ @RobertF (and Frank as well): How exactly would you use "Bayesian hierarchical/penalized regression" instead of ANOVA with post-hoc multiple comparisons? Say it's one-way ANOVA with 5 groups corresponding to 5 different treatments. What is hierarchical about it? $\endgroup$
    – amoeba
    Mar 4, 2016 at 21:49
  • $\begingroup$ @amoeba - No, the model wouldn't be hierarchical for a 5 treatment one-way ANOVA unless, for example, the data contains multiple member-level records. However a simple lasso or ridge regression could still be performed to adjust effect sizes. $\endgroup$
    – RobertF
    Mar 5, 2016 at 3:43
1
$\begingroup$

My first thought after reading your question is to use a penalized regression to adjust effect sizes in multiple comparisons. Since an ANOVA is a form of linear regression with multiple categorical variables, one of the penalized regression techniques (such as a lasso, ridge regression, or elastic net) can be applied to adjust the effect sizes of each comparison (= regression coefficients) towards the group mean. Comparisons which are not "statistically significant" result in the effect size being reduced to zero.

Improve this answer
$\endgroup$
0
$\begingroup$

You generally wouldn't adjust the estimate of the effect size itself. However, you can adjust the widths of the confidence intervals around the effect size, by using a procedure such as Bonferroni to produce "simultaneous confidence intervals."

Using the Bonferroni procedure, the alpha level for each individual test is set at the desired overall alpha level divided by m (where m is the number of tests). Thus, to adjust confidence intervals when the desired overall confidence level is 1 - alpha, simply compute each individual confidence interval at 1 - alpha/m.

Not all multiple comparisons procedures have corresponding procedures for simultaneous confidence intervals like Bonferroni does, but many do.

Improve this answer
$\endgroup$
-1
$\begingroup$

I disagree with the previous response... The protection afforded by a multiple comparison procedure like bonferroni is to limit the likelihood of making an error (in this case a Type I error). An effect size measure should only be computed on a significant result (where the null hypothesis has been rejected) - in that case you are no longer working in the space of potentially making an error, you are now working in a space of nominal truth - you have to assume the null to be false in order to do the EF calculation meaningfully. So, you should have arrived at that nominal truth conservatively (with a protected alpha) but once you get there you are free to compute away....

Improve this answer
$\endgroup$
2
  • $\begingroup$ There was no "mod's response" here. If you refer to Bjorn's comment, then he is not a moderator. $\endgroup$
    – amoeba
    Mar 4, 2016 at 18:37
  • 1
    $\begingroup$ Using statistical significance to decide which effects to examine is not correct. Use confidence intervals and point estimates regardless of P-values. $\endgroup$
    – Frank Harrell
    Mar 4, 2016 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.