1
$\begingroup$

Let $f(x|\rho)$ be the Bernoulli pmf with probability $\rho$ of success. \begin{align} f(x|\rho) = \left\{ \begin{array}{ll} \rho & x = 1 \\ 1-\rho & x=0 \end{array} \right. \end{align}

The problem I'm interested in.

Process $1$ generates successes and failures by randomly sampling from $f(x|\rho_1)$, and process $2$ generates successes and failures by randomly sampling from $f(x|\rho_2)$.

For the sake of concreteness, process 1 consists of a student learning a concept in a particular way, solving a problem that requires them to apply that concept immediately afterward, and either getting it right (resulting in a score $x = 1$) with probability $\rho_1$ or wrong ( resulting in a score $x = 0$) with probability $1-\rho_1$. Process 2 consists of a student learning the same concept in a different way, solving the same application problem, and either getting it right or wrong with probability $\rho_2$ and $1-\rho_2$ respectively.

Now suppose that we separate a class of students into two equal-sized groups with half undergoing the first treatment and half undergoing the second, and the inference problem we care about is of the following flavor:

Given scores $x_{11}, x_{12}, \dots, x_{1n}$ from the students in treatment 1 and scores $x_{21}, x_{22}, \dots, x_{2n}$ from students in treatment 2, define and infer the value of some parameter $\delta$ that can be interpreted as a "standardized difference" in the parameters $\rho_2$ and $\rho_1$.

An analog that suggests a solution.

If instead of being drawn from a Bernoulli distribution, each student's score in treatment 1 were drawn from a normal distribution with mean $\mu_1$ and variance $\sigma_1^2$, and if each student's score in treatment 2 were drawn from a normal distribution with mean $\mu_2$ and variance $\sigma_2^2$, then I would be tempted to define the following parameter as a measure of the ``effect size'' between these two treatments: $$ \delta_\mathrm{norm} = \frac{\mu_1 - \mu_2}{\sqrt{\frac{1}{2}(\sigma_1^2+\sigma_2^2)}} $$ because if I were two then generate samples of size $n$ from both of these treatments, I could use Cohen's $d$, a common effect size statistic which in this case would be: \begin{align} d = \frac{\bar x_2 - \bar x_1}{\sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}}} = \frac{\bar x_2 - \bar x_1}{\sqrt{\frac{(n-1)s_1^2 + (n-1)s_2^2}{n+n-2}}} = \frac{\bar x_2 - \bar x_1}{\sqrt{\frac{1}{2}(s_1^2 + s_2^2)}} \end{align} to accurately estimate $\delta$. This gives me confidence that $\delta$ is a parameter that coincides with a commonly used effect size statistic.

Back to the problem of interest.

For the problem of interest, working by analogy, I'm tempted to use the fact that the mean of the Bernoulli pmf is $\rho$ and its variance is $\rho(1-\rho)$ to define the following effect size parameter in the case of student scores drawn from Bernoulli: \begin{align} \delta_\mathrm{Bern} = \frac{\rho_2 - \rho_1}{\sqrt{\frac{1}{2}\big(\rho_1(1-\rho_1) + \rho_2(1-\rho_2)\big)}} \end{align} and this could be estimated by plugging in $\hat \rho_1$ and $\hat \rho_2$, the experimentally observed proportion of successes for each group of students, for $\rho_1$ and $\rho_1$.

However, I haven't encountered this solution to the problem at hand anywhere in the statistics literature. It's appealing to me because it has a nice interpretation: it's the difference in successes of each of the two groups normalized by a number that represents a kind of weighted combination of the variations one would expect if a student were to take a test with many questions, each having a probability $\rho_1$ or $\rho_2$ of being answered correctly depending on the treatment.

Questions.

Is the parameter $\delta_\mathrm{Bern}$ I've defined above commonly used as an effect size parameter for this sort of problem? Is there a more commonly-used and/or better way to attack this? Will I confuse practitioners if I use an estimate of $\delta_\mathrm{Bern}$ to compute effect size in such an experiment?

$\endgroup$
1
$\begingroup$

This is just an extended comment.

If the effect size metric makes sense for your application, then your next step is to see if you can get consensus for its use.

The issue (in my mind) about effect sizes is how to determine how big is big. That depends on the subject matter area AND the particular application. Indiscriminant use of Cohen's small, medium, and large categories outside of the original application is at best not good practice.

For your effect size metric one can graphically show "all possible values" and that might suggest for your application what is small, medium, and large and maybe even suggest modifications in the effect size metric.

delta Bern

$\endgroup$
  • $\begingroup$ Thanks for the comments. The squares of the variances is an unintentional error that I've now fixed. Thanks for pointing that out. The contour plot is also a conceptually useful way of visualizing the parameter that I hadn't thought of, so thanks for that as well. $\endgroup$ – joshphysics Dec 15 '18 at 10:30
  • $\begingroup$ I've modified the figure to reflect the updated effect size metric. $\endgroup$ – JimB Dec 15 '18 at 14:46
  • $\begingroup$ Thanks. The more I read about effect size (including your comments), the more I'm convinced that when a researcher computes one, they should explicitly comment on its interpretation (especially interpreting its magnitude) in the context being studied, but this doesn't seem to be common in, say, the behavioral and/or psychological sciences literature. Your contour plot has also got me thinking about whether or not it "makes sense" to have an effect size measure with non-linear level curves. It also perturbs me that many practitioners seem to be using Cohen's $d$ without much thought. $\endgroup$ – joshphysics Dec 15 '18 at 18:48
  • $\begingroup$ A contour plot for $\rho_1-\rho_2$ doesn't look too much different from that of $\delta_{Bern}$ in the sense that the parameter space is ordered in a somewhat similar manner. So you might want to consider what $\delta_{Bern}$ offers that $\rho_1-\rho_2$ doesn't. (That's a subject matter issue rather than a statistical issue.) You could even start by drawing contour lines of equal effect size and then figure out a formula that would get you those contours (at least approximately) that order the parameter space for your particular application. $\endgroup$ – JimB Dec 16 '18 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.