2
$\begingroup$

Pardon a relative novice's question. I'm seeking a reference that describes, compares, and gives formulas for the standard error for ratios and the standard error for differences between means when small Ns are involved.

Here is the situation at hand:
- Multiple samples from two independent populations.
- The underlying variable is normally distributed in each population, but with somewhat different means and standard deviations.
- At least one sample will always be small, perhaps 3 to 15 cases, the other sample will generally be 50 or 100 cases, or even larger.
- Statistics of interest: ratio of means and difference between means.
- The variables are grades on multiple choice tests and such tests tend to be normally distributed. The p (i.e., probability of getting a question correct, AKA difficulty level) for the tests in question is around .7

Question: How will the standard errors of these two sample statistics compare?

Update based on comments

The specific question I have is whether the mean standardized difference between 2 groups will be a more or less stable measure (i.e., have a larger or smaller standard error) than the ratio of the means, when at least one of the groups is small.

$\endgroup$
13
  • $\begingroup$ One reason you haven't gotten any responses is that there is no universal way in which those SEs compare. They're not even on the same scales! You are asking us to compare proverbial apples and oranges. Could you reformulate your question more specifically? $\endgroup$
    – whuber
    Jun 27 '19 at 13:02
  • $\begingroup$ @whuber I added the nature of the variables (M/C tests). Is that enough or do you need more detail? $\endgroup$
    – Joel W.
    Jun 28 '19 at 15:28
  • $\begingroup$ It's close--but the answer depends on how the grading is done. You can get different answers depending on whether you are reporting raw grades or standardizing them to some scale. The basic question remains: what sense can it possibly make to compare a ratio of two comparable quantities--which is unitless--to one of those quantities, which has a unit of measurement?? One is led to suspect you are not adequately formulating some underlying problem, which raises concerns that any answers to this question will be (at best) misleading. $\endgroup$
    – whuber
    Jun 28 '19 at 15:31
  • $\begingroup$ Why do you wish to compare the standard errors? In the end, what are you really trying to assess? Are you going to perform a hypothesis test and need to decide whether to formulate it in terms of either a ratio of means or differences? $\endgroup$
    – jsk
    Jun 28 '19 at 17:30
  • 1
    $\begingroup$ @jsk Good question. On reflection I may need to reframe the question. For now, I would to be able to compare the two groups with respect to how similar or different they are in terms of central tendency (mean scores). $\endgroup$
    – Joel W.
    Jun 28 '19 at 19:30
1
+100
$\begingroup$

Standard error of ratio of two approximately normal distributed variables

The standard error relates to the variance or standard deviation and because of this your question might be related to some other questions on stackexchange:

  • What are the mean and variance of the ratio of two normal variables, with non-zero means?

    The variance of the ratio of two normally distributed variables does not exist. It is undefined. This is because the reciprocal of a normal distributed variable has no defined mean and variance.

  • estimation of population ratio using delta method

    However, in practice variables are not always normal distributed (the normal distribution is just an approximation). For instance, in many cases the domain of the denominator has some lower bound that is positive (unlike the normal distribution). In that case the ratio will have finite moments.

    You can estimate the variance using the delta method (see for example estimation of population ratio using delta method). In that stackexchange link the ratio is approximated by the first terms in a Taylor series (which requires $\sigma << \mu$, ie. small errors) : $$Z = \frac{X}{Y} \approx \frac{\mu_X}{\mu_X} + (X-\mu_X) \cdot \frac{1}{\mu_Y} + (Y-\mu_Y) \cdot \frac{-\mu_X}{\mu_Y^2} $$ and the variance (if it exists) is: $$\sigma_{X/Y}^2 \approx \sigma_{X}^2 \cdot \frac{1}{\mu_Y^2} + \sigma_{Y}^2 \cdot \frac{\mu_X^2}{\mu_Y^4}$$

(see also https://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables)


So those are some few thoughts about the expression of the standard error for a ratio. But your more direct question: "How will the standard errors of these two sample statistics compare?" is a bit unclear to me.

We can say very straightforward that $$\sigma_{X-Y}^2 = \sigma_{X}^2+\sigma_{Y}^2$$ and $$\sigma_{X/Y}^2 \approx \sigma_{X}^2 \cdot 1/\mu_Y^2 + \sigma_{Y}^2 \cdot {\mu_X^2/\mu_Y^4}$$

But is that what you are looking for? In what sense do you wish to compare the standard errors? Do you wish to evaluate which statistic is the best in order to test the hypothesis Y=X?


Stability

Below is an example for the variables $$\begin{array}{rcl} Z_1 &=& \frac{\bar{X}_n}{\bar{Y}_n} \\ Z_2 &=& {\bar{X}_n}-{\bar{Y}_n}\end{array}$$ with $$\begin{array}{cccc}\bar{X}_n &=& \frac{1}{n} (X_1, X_2, ... , X_n) & \qquad X_i \sim N(\mu_X=1,\sigma_X = 2) \\ \bar{Y}_n &=& \frac{1}{n} (Y_1, Y_2, ... , Y_n)& \qquad Y_i \sim N(\mu_Y=1,\sigma_Y = 2) \end{array}$$

$X_n,Y_n$ can be considered as a sort of random walk and as $n \to \infty$ you will get $X_n,Y_n \to \mu_X, \mu_Y$. You can see the distribution of $X_n,Y_n$ as a multivariate Gaussian that becomes more and more concentrated around the point $\mu_X, \mu_Y$ and this is like sort of zooming in to the point $\mu_X, \mu_Y$.

The distribution for the values of $Z_1$ and $Z_2$ can be imagined by superimposing to this random walk the isolines for values of $Z_1$ and $Z_2$. The lines for $Z_2$ are parallel and no matter what value of $n$ the statistic $Z_2$ is normal distributed. The lines for $Z_1$ are not parallel and the distribution for $Z_1$ is not normal distributed, but when you zoom in (when $n$ increases) the lines are more parallel, and the distribution of $Z_1$ becomes more and more normal distributed. In the case of $\mu_X=\mu_Y=1$ you will even get that $Z_1$ and $Z_2$ become as close as you wish for sufficiently large $n$.

example

  • Note while the distribution for $\bar{X}_n/\bar{Y}_n$ is asymptotically normal distributed for $n \to \infty$ (see also https://stats.stackexchange.com/a/399952/164061 ) it will still have some sort of Cauchy/Lorentz distribution component and an expression of stability in terms the standard deviation (or error) is not defined.

    But the good news is that in practice we often do not have truly a normal distribution. So when your grades are always positive (no negative and zero values as the normal distribution would technically include as well) then the ratio will have a defined and finite deviation/variance.

  • Note that the image above has $Z_1$ and $Z_2$ become equal for large $n$. But this will only hold for $\mu_X=\mu_Y=1$. You will not in general get the same sample variance for $Z_1$ and $Z_2$, this is even true when $\mu_X = \mu_Y$.

    In the image you can use the distance between the iso-lines as a measure for the standard deviation. When the lines are close to each other then changes in $X_n,Y_n$ will make large differences in the statistic. When the lines are far from each other then changes in $X_n,Y_n$ will make small differences in the statistic.

    You can see that neither $X/Y$ or $X-Y$ is the best in every region. $X/Y$ will 'perform' better when $X,Y$ are large and $X-Y$ will 'perform' better when $X,Y$ are small. If you would draw some boundary lines for a hypothesis test like $a < X/Y < b$ and $c < X-Y < d$, then the regions will be different geometries. The statistical test $a < X/Y < b$ will relate to a region in the shape of a cone. The statistical test $c < X-Y < d$ will relate to a region in the shape of a diagonal bar. These regions will not reject/accept all cases the same. One region will be more strict close to the origin, the other will be more strict far away from the origin.

$\endgroup$
13
  • $\begingroup$ This is helpful. I would like to see these evaluated for small sample sizes. The variables are grades on multiple choice tests and such tests tend to be normally distributed. The p for the tests in question is around .7 $\endgroup$
    – Joel W.
    Jun 28 '19 at 15:25
  • $\begingroup$ @JoelW. 1 what do you mean by 'evaluated for small sample sizes'? Do you take a sample of multiple fractions? The formula I give above is for the ratio of two variables (e.g. the ratio of two sample means, you will use the variance/deviation of those two variables, that will be the sample distribution not the population distribution) 2 What does 'the p' mean? 3 Also, you still need to clarify what you are looking for. What 'comparison' do you want to make? There is no way to answer how they compare when you do not specify what you mean by a comparison. $\endgroup$ Jun 29 '19 at 17:02
  • $\begingroup$ By "evaluated for small samples" I mean to ask: what numeric values will the statistics have when the size of one of the two samples is small (perhaps 2 to 15) and the other sample is say 10 times as large. This type of sampling is repeated around the country often in practical settings. I do not plan to personally be collecting such samples. $\endgroup$
    – Joel W.
    Jul 1 '19 at 12:23
  • $\begingroup$ @JoelW. can you provide some data? You are asking now 'what numeric values will the statistics have'. This can not be answered without data or more information. What is your input that you wish to evaluate? $\endgroup$ Jul 1 '19 at 12:42
  • $\begingroup$ You say, "𝑋−𝑌 will 'perform' better when 𝑋,𝑌 are small". What do you mean by small (and large). $\endgroup$
    – Joel W.
    Jul 3 '19 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.