6
$\begingroup$

I encountered a weird effect when computing eta squared in ANOVA. Here is a short simulation to demonstrate it.

I simulate $k$ groups with $n=10$ each, with all values drawn from standard normal distribution (i.e. under null hypothesis of no difference between groups). I conduct a one-way ANOVA to compare these groups, and compute eta squared $\eta^2 = \mathrm{SS}_\mathrm{btw}/\mathrm{SS}_\mathrm{tot}$ as a measure of effect size. I then plot the histogram of resulting values over many simulations. Here is the figure for $k=2,5,10$:

Distribution of eta squared in one-way ANOVA under null hypothesis

One can see that whereas at $k=2$ the distribution peaks at zero, with more groups the distribution starts to peak at a non-zero positive value. I have a number of questions about this phenomenon.

  1. I did some googling (and searched this forum) and see that eta squared is often called a "biased" measure of effect size. Does this refer exactly to what I stumbled upon?

    Update: no. As @gung and @Silverfish clarified below, eta squared is trivially biased because it is constrained to be positive, and so $\mathbb{E}(\eta^2) \ne 0$ under the null, meaning that it is by definition biased. I am talking about a different phenomenon that does not seem to have a name (?), so I will call it "mode-biased-under-null".

  2. Under what conditions does this mode-bias-under-null appear? I cannot get it with a t-test ($k=2$), both with equal and non-equal sample sizes, and seem to fail to get it with $k=3$. What is a condition (in terms of $k$ and $n_i$) that guarantees that the mode will be at zero?

    Update: it seems that the answer is $k \le 3$. Please see discussion here: What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?

  3. What are the alternatives to eta squared that do not suffer from this weird mode-biadness-under-null?

    I would like to have a measure of effect size (informally: measure of "separability" between groups) that:

    • is unbiased can be biased, but should peak at zero under the null hypothesis;
    • is between $0$ and $1$ so that $0$ can be interpreted as full overlap between groups and $1$ as non-overlapping distributions of groups (i.e. $100\%$ classification accuracy by a trivial algorithm);
    • does not have to be given by a simple formula, as long as it can be computed (e.g. for $k=2$ an area under a ROC curve would be an option);
    • works with unbalanced groups;
    • generalizes to n-way ANOVA with several factors, as eta squared generalizes to partial eta squared (but no nesting, repeated measures, or other nasty ANOVA designs are of interest to me; just e.g. two factors with an interaction).

Update: why would I care about it?

In my field (neuroscience) people often test a whole bunch of DVs (activity of single neurons) for dependency on some categorical IVs. Often this is done when there is only one categorical IV of interest, and it is binary. In this case, a histogram of effect sizes is often plotted across a population of neurons. Here is one example from this Nature paper:

Kepecs et al. histogram

Here $563$ neurons were tested, $136$ showed significant difference at $p<0.05$ level, and effect size ("outcome preference") was computed as an appropriately scaled area under the ROC curve.

I want to make a similar histogram, but when neurons are tested for tuning not to a binary factor, but to a multilevel factor. So I was going to run ANOVA and use $\eta^2$ as the effect size (or perhaps signed $\eta$, as my factor is in fact ordinal, so a meaningful sign can be attached to $\sqrt{\eta^2}$), but the resulting histogram does not peak at zero (and in case of signed $\eta$ it is bimodal), which will definitely confuse all the readers.

$\endgroup$
11
  • 2
    $\begingroup$ "is unbiased (peaks at zero under null hypothesis)": I may be missing something here, but relating this back to multiple regression, it's almost impossible for $R^2 = 0$ even under $H_0$. You'd have to get lucky with your $X$ being orthogonal to observed $Y$. Demanding that $\mathbb{E}(R^2) = 0$ under the null is obviously an unreasonable requirement because it is almost always going to be positive, but never negative (similar but not identical considerations for $R^2_{adj}$). As $\rho^2$ is zero under the null, $R^2$ is unsurprisingly biased. But perhaps you mean something else by "biased"? $\endgroup$ – Silverfish Dec 22 '14 at 23:42
  • $\begingroup$ @Silverfish, I am not very familiar with this topic, so it might very well be that my questions are naive. Sorry for that and thanks for understanding. But I am confused. Is $\eta^2 = R^2$ going to peak at zero for a t-test? If yes, then how does it relate to what you wrote? Or should I maybe be thinking about $r$ and not about $r^2$? Wait, is then the problem that for many groups / many predictors there is no way of "taking a square root" from $R^2$ and assigning it a meaningful sign? $\endgroup$ – amoeba Dec 22 '14 at 23:48
  • $\begingroup$ I'm afraid every time I see an ANOVA I default to thinking about a regression! Users whose brains work in "ANOVA mode" are better-placed to answer this question than I am, but $R^2$ and $\eta^2$ are closely linked. $\endgroup$ – Silverfish Dec 22 '14 at 23:54
  • $\begingroup$ Yes, I believe $\eta^2$ is $R^2$. So let's talk about regression then; as far as I understand, a t-test corresponds to a regression with only one predictor. It is obvious that $r$ (correlation coefficient) will peak at zero under null. But what will be the mode of $r^2$ distribution under the null? If it is also zero, then one part of my question is: why is this different in multiple regression? Note that I am not asking about $\mathbb{E}(R^2)$, as you wrote above; I am asking about the mode. $\endgroup$ – amoeba Dec 23 '14 at 0:01
  • 1
    $\begingroup$ When I see "$\hat{\theta}$ is biased", I think this means $\mathbb{E}(\hat{\theta})\neq\theta$. Regarding $R^2$ as an estimator of the % variation explained in the population by the correct model, or $R$ as an estimator of the correlation between fitted and observed values in the population, the concept of them being "biased" estimators does make sense. However it relates to mean not mode (which is why I wonder if you're looking for a different word than "bias") and it's "obvious" they're biased if $H_0$ is true: since they're positive they can't average out to zero. $\endgroup$ – Silverfish Dec 23 '14 at 0:03
5
$\begingroup$

$\eta^2$ is the same as $R^2$ in a one-way ANOVA. It is bounded by $[0,\ 1]$. When the null hypothesis holds, the true value of $\eta^2$ is $0$. So the estimator $SSB/SST$ must be biased unless either it can only return $0$ when the null hypothesis is true, or if half its distribution is $<0$. Since it cannot be $<0$, and it can yield non-zero values, even when the null obtains, it must be biased. On the other hand, it is consistent, in the sense that $\eta^2\rightarrow 0$ as $N$ goes to infinity when the null holds.

$\endgroup$
4
  • $\begingroup$ Thanks, gung. Your answer and @Silverfish'es comments above made it clear and obvious that $\eta^2$ is biased for a trivial reason of being constrained to be positive. If one chooses any way of assigning sign to $\pm \sqrt{\eta^2}$, then such $\eta$ will be symmetrically distributed around zero under $H_0$ and hence unbiased. However, what concerns me is that distribution of $\eta^2$ has a non-zero mode for $k>3$ groups; therefore the distribution of my $\eta$ will be bimodal. My actual question is if there is any measure of effect size without this weird property. I will edit to clarify. $\endgroup$ – amoeba Dec 23 '14 at 11:11
  • $\begingroup$ @amoeba In fact the sign on $R$, i.e. $\eta$, is not arbitrary, as I explained here! $R$ is the correlation between the fitted and observed values of the dependent variable. $\endgroup$ – Silverfish Dec 23 '14 at 12:00
  • $\begingroup$ What you essentially explain there is that the sign is always positive. I would argue that in some cases there can be more meaningful conventions. I commented there. $\endgroup$ – amoeba Dec 23 '14 at 12:15
  • $\begingroup$ @gung: I edited my question to remove my confusion about "biasedness" and to add an explanation of why I care about the mode of the distribution of effect sizes. $\endgroup$ – amoeba Dec 23 '14 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.