12
$\begingroup$

Related to Analysing ratios of variables and How to parameterize the ratio of two normally distributed variables, or the inverse of one?.

Suppose I have a number of samples from four different continous random distributions, all of which we can assume to be roughly normal. In my case, these correspond to some performance metrics of two different filesystems (say, ext4 and XFS), both with and without encryption. The metric might be, for example, the number of files created per second, or the average latency for some file operation. We can assume that all samples drawn from these distributions will always be strictly positive. Let's call these distributions $\textrm{Perf}_{fstype,encryption}$ where $fstype \in \{xfs,ext4\}$ and $encryption \in \{crypto,nocrypto\}$.

Now, my hypothesis is that encryption slows down one of the filesystems by a bigger factor than the other. Is there some simple test for the hypothesis $\frac{E[\textrm{Perf}_{xfs,crypto}]}{E[\textrm{Perf}_{xfs,nocrypto}]} < \frac{E[\textrm{Perf}_{ext4,crypto}]}{E[\textrm{Perf}_{ext4,nocrypto}]}$?

Improve this question
$\endgroup$
4
  • $\begingroup$ Some text seems to have been deleted from the middle of this question. Do you think you could restore it? $\endgroup$ – whuber♦ Feb 20 '12 at 14:55
  • $\begingroup$ I think the "Thus, the" was left there by mistake, at least I cannot think what I would have wanted to add to that. Probably it was something that I finally moved to the second paragraph. $\endgroup$ – Sami Liedes Feb 20 '12 at 15:06
  • $\begingroup$ You could fit a generalized linear model for a normal distribution with a log link function. $\endgroup$ – onestop Feb 20 '12 at 20:31
  • 1
    $\begingroup$ "Number of files" and "average latency" cannot be normally distributed (neither can be negative for a start). Both are likely to be somewhat right skew. Number of files is a discrete count. $\endgroup$ – Glen_b Jan 20 '16 at 22:47
13
$\begingroup$

One alternative to StasK's fine answer is to use a permutation test. The first step is to define a test statistic $T$, perhaps:

$T = \frac{\widehat{Perf}_{ext4,crypto}}{\widehat{Perf}_{ext4,nocrypto}} - \frac{\widehat{Perf}_{xfs,crypto}}{\widehat{Perf}_{xfs,nocrypto}}$

where $\widehat{Perf}_{ext4,crypto}$ is, perhaps, the sample mean of the observations of $\text{Perf}_{ext4,crypto}$, etc. (This fits with your definition of the hypothesis as the ratio of the expectations rather than the alternative possibility of the expectation of the ratio - which alternative may be what you really want.) The second step is to randomly permute the labels $ext4, \space xfs$ in the data many times, say, $i=1, \dots, 10000$, and calculate $T_i$ for each permutation. The final step is to compare your original $T$ with the observed $T_i$; the permutation-estimated p-value would be the fraction of the $T_i \leq T$.

The permutation test frees you from reliance on asymptotics, but of course depending upon your sample size (and the data too, of course), the delta method, which I use occasionally also, may work just fine.

Improve this answer
$\endgroup$
7
  • $\begingroup$ That's a good suggestion, as well! $\endgroup$ – StasK Feb 20 '12 at 20:01
  • $\begingroup$ Note that the ratio of two centred normal variables is a Cauchy variable. $\endgroup$ – Xi'an Feb 20 '12 at 21:06
  • 1
    $\begingroup$ @Xi'an: Is it clear we can assume they are independent here? As you know, that would be necessary for this result to hold (and have a chance at being useful). $\endgroup$ – cardinal Feb 20 '12 at 23:18
  • $\begingroup$ @cardinal: yes, indeed, they would have to be independent! $\endgroup$ – Xi'an Feb 21 '12 at 5:33
  • 1
    $\begingroup$ As a very snobbish technical point -- permutation works a little better when your test statistic is pivotal / does not involve unknown parameters / is variance-stabilized... at least under the null. With proportions, you can do an arc sin transformation. With strictly positive continuous quantities, I would probably start with logs. But this is really icing on the cake. $\endgroup$ – StasK Apr 5 '19 at 17:15
4
$\begingroup$

You can compute the (asymptotic) standard error of the ratio using the delta-method. If you have two random variables $X$ and $Y$ such that $$\sqrt{n}\left(\begin{array}{c} \bar X-\mu_X \\ \bar Y-\mu_Y\end{array}\right) \rightarrow N\left( \left( \begin{array}{c} 0 \\ 0 \end{array}\right), \left( \begin{array}{cc} \sigma_{XX} & \sigma_{XY} \\ \sigma_{XY} & \sigma_{YY} \end{array} \right) \right) $$ in distribution (which would be the case if you have independent data, but it would also hold in a more general case of clustered data when you ran your tests on different machines), then for the ratio $r=\bar Y/\bar X$ with the population analogue of $r_o = \mu_Y/\mu_X$, we have $$ \sqrt{n}(r-r_0) \to N(0,\frac{\mu_Y^2}{\mu_X^4}\sigma_{XX} - 2\frac{\mu_Y}{\mu_X^3}\sigma_{XY} + \frac1{\mu_X^2}\sigma_{YY}) $$ If $X$ and $Y$ are independent, as might be reasonable to assume in your case, then this expression simplifies somewhat by dropping $\sigma_{XY}$, so we get that the squared coefficients of variations sum up: $${\rm CV}^2[r] = {\rm CV}^2[\bar X] + {\rm CV}^2[\bar Y]$$ It has the additional advantage that the sample sizes might be different. Furthermore, if your RHS and LHS are independent, you can form the $z$-test statistic for $H_0:$ no difference by taking the difference of the ratios and dividing it by the corresponding standard error obtained from these CV's.

I hope you can take it from there and perform the remaining back of the envelope calculations to obtain the final formula.

Note that the result is asymptotic, and the ratio $r$ is a biased estimator of $r_0$ in small samples. The bias has the order of $O(1/n)$, and disappears asymptotically when compared to sampling variability which is of the order $O(1/\sqrt{n})$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the excellent and enlightening answer! I think I'll pick jbowban's permutation test for my studies because I think I understand it and its limitations better, but the delta method definitely looks like something I need to study and figure out. $\endgroup$ – Sami Liedes Feb 21 '12 at 15:31
  • $\begingroup$ @stask could this be donehere ? stats.stackexchange.com/questions/398436/… $\endgroup$ – Xavier Bourret Sicotte Mar 20 '19 at 21:10
  • $\begingroup$ Xavier, I think @usεr11852 gave a good answer. I won't bother adding to that. $\endgroup$ – StasK Mar 21 '19 at 18:04
  • $\begingroup$ @StasK - under what conditions are the conditions you state in your answer valid ? Is the convergence of the ratio statistic guaranteed by the previous assumption and the Delta method ? $\endgroup$ – Xavier Bourret Sicotte Mar 26 '19 at 21:17
  • $\begingroup$ It's asymptotics... nothing is ever guaranteed, and error boundaries are difficult to impossible to get. All the delta method (or any other weak convergence result) is saying is that as you increase the sample size, the difference between the actual finite sample distribution from the asymptotic distribution will get smaller. That may mean that as you increase the sample size from 1000 to 10000, the vertical difference between cdfs would go down from 0.2 to 0.1, and the latter is still unacceptable for practical purposes. Or it may mean that the difference goes from 0.01 to 0.001. $\endgroup$ – StasK Apr 5 '19 at 17:23
0
$\begingroup$

The ratio of Normal variates is distributed Cauchy. Knowing that, you can simply perform a Bayes Factor Test.

This was a rather spontaneous idea. I am now unsure about the data-generating mechanism. Do you install different file systems on the same PC and then benchmark for the two cases, so that we can assume a hierarchical data structure?

Also I am not sure looking ratios actually makes sense.

And then you wrote the ratio of the expected values, whereas I thought of the expected value of the ratios. I guess I need more information about the data generation before moving on.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ The ratio of normals is only Cauchy if (a) they are independent and (b) have the same variance. $\endgroup$ – cardinal Feb 20 '12 at 20:16
  • $\begingroup$ Xi'an had the same thought I guess... $\endgroup$ – joint_p Feb 20 '12 at 23:08
  • 1
    $\begingroup$ It is not clear (to me, at least) that any such independence structure exists or that they will have zero mean. Perhaps, if you can expand on your answer it will help make clearer the approach you are suggesting. :) $\endgroup$ – cardinal Feb 20 '12 at 23:23
  • 1
    $\begingroup$ @cardinal - I thought it was a ratio of independent normals with zero mean was cauchy with zero median and scale parameter equal to ratio of the normal standard deviations. If they have non zero mean then it is not cauchy. $\endgroup$ – probabilityislogic Feb 21 '12 at 8:18
  • $\begingroup$ @prob: (+1) You are right! Thanks for catching that. I dropped the "standard" and "zero mean" in my first comment (the latter did manage to make it into my second one). $\endgroup$ – cardinal Feb 21 '12 at 12:43
0
$\begingroup$

In cases where you cannot perform permutations, for example when the sample size creates millions of possibilities, another solution would be Monte Carlo resampling.

The null hypothesis is that there is no difference in the speed between $ext4$ and $xfs$, for $nocrypto$ and $crypto$. Therefore, the average ratio $\frac{ext4}{xfs}$ of all $nocrypto$ samples does not differ from that of $crypto$.

$H_{0}:T_{observed}=\frac{\sum x_{nocrypto} }{n_{nocrypto}}-\frac{\sum x_{crypto} }{n_{crypto}}=0 $

where $ x=\frac{ext4}{xfs} $

and $ n=sample\, size $

If $H_{0}$ is true, randomly picking results for ratios of $nocrypto$ or $crypto$ would also result in $T_{observed}=0$. One would calculate:

$ T_{resampling}=\frac{x_{1}^{random}{+ x}_{n}^{random}}{n_{nocrypto}}-\frac{x_{1}^{random}{+ x}_{n}^{random}}{n_{crypto}} $

and perform, say, 10,000 rounds of resampling. The resulting distribution of $T_{resampling}$ values is the confidence interval for $H_{0}$. The difference between $nocrypto$ and $crypto$ ratio is significant if the calculated $T_{observed}$ value lies outside the range of, e.g., 95% $(p < 0.05)$ of the $T_{resampling}$ values.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.