Test for significant difference in ratios of normally distributed random variables Related to Analysing ratios of variables and How to parameterize the ratio of two normally distributed variables, or the inverse of one?.
Suppose I have a number of samples from four different continous random distributions, all of which we can assume to be roughly normal. In my case, these correspond to some performance metrics of two different filesystems (say, ext4 and XFS), both with and without encryption. The metric might be, for example, the number of files created per second, or the average latency for some file operation. We can assume that all samples drawn from these distributions will always be strictly positive. Let's call these distributions $\textrm{Perf}_{fstype,encryption}$ where $fstype \in \{xfs,ext4\}$ and $encryption \in \{crypto,nocrypto\}$.
Now, my hypothesis is that encryption slows down one of the filesystems by a bigger factor than the other. Is there some simple test for the hypothesis $\frac{E[\textrm{Perf}_{xfs,crypto}]}{E[\textrm{Perf}_{xfs,nocrypto}]} < \frac{E[\textrm{Perf}_{ext4,crypto}]}{E[\textrm{Perf}_{ext4,nocrypto}]}$?
 A: You can compute the (asymptotic) standard error of the ratio using the delta-method. If you have two random variables $X$ and $Y$ such that 
$$\sqrt{n}\left(\begin{array}{c} \bar X-\mu_X \\ \bar Y-\mu_Y\end{array}\right) \rightarrow N\left( \left( \begin{array}{c} 0 \\ 0 \end{array}\right), \left( \begin{array}{cc} \sigma_{XX} & \sigma_{XY} \\ \sigma_{XY} & \sigma_{YY} \end{array} \right) \right) $$ in distribution (which would be the case if you have independent data, but it would also hold in a more general case of clustered data when you ran your tests on different machines), then for the ratio $r=\bar Y/\bar X$ with the population analogue of $r_o = \mu_Y/\mu_X$, we have
$$
\sqrt{n}(r-r_0) \to N(0,\frac{\mu_Y^2}{\mu_X^4}\sigma_{XX} - 2\frac{\mu_Y}{\mu_X^3}\sigma_{XY} + \frac1{\mu_X^2}\sigma_{YY})
$$
If $X$ and $Y$ are independent, as might be reasonable to assume in your case, then this expression simplifies somewhat by dropping $\sigma_{XY}$, so we get that the squared coefficients of variations sum up:
$${\rm CV}^2[r] = {\rm CV}^2[\bar X] + {\rm CV}^2[\bar Y]$$
It has the additional advantage that the sample sizes might be different. Furthermore, if your RHS and LHS are independent, you can form the $z$-test statistic for $H_0:$ no difference by taking the difference of the ratios and dividing it by the corresponding standard error obtained from these CV's.
I hope you can take it from there and perform the remaining back of the envelope calculations to obtain the final formula.
Note that the result is asymptotic, and the ratio $r$ is a biased estimator of $r_0$ in small samples. The bias has the order of $O(1/n)$, and disappears asymptotically when compared to sampling variability which is of the order $O(1/\sqrt{n})$.
A: One alternative to StasK's fine answer is to use a permutation test.  The first step is to define a test statistic $T$, perhaps:
$T = \frac{\widehat{Perf}_{ext4,crypto}}{\widehat{Perf}_{ext4,nocrypto}} - \frac{\widehat{Perf}_{xfs,crypto}}{\widehat{Perf}_{xfs,nocrypto}}$
where $\widehat{Perf}_{ext4,crypto}$ is, perhaps, the sample mean of the observations of  $\text{Perf}_{ext4,crypto}$, etc.  (This fits with your definition of the hypothesis as the ratio of the expectations rather than the alternative possibility of the expectation of the ratio - which alternative may be what you really want.)  The second step is to randomly permute the labels $ext4, \space xfs$ in the data many times, say, $i=1, \dots, 10000$, and calculate $T_i$ for each permutation.  The final step is to compare your original $T$ with the observed $T_i$; the permutation-estimated p-value would be the fraction of the $T_i \leq T$.  
The permutation test frees you from reliance on asymptotics, but of course depending upon your sample size (and the data too, of course), the delta method, which I use occasionally also, may work just fine.
A: The ratio of Normal variates is distributed Cauchy. Knowing that, you can simply perform a Bayes Factor Test.
This was a rather spontaneous idea.
I am now unsure about the data-generating mechanism. 
Do you install different file systems on the same PC and then benchmark for the two cases, so that we can assume a hierarchical data structure? 
Also I am not sure looking ratios actually makes sense.
And then you wrote the ratio of the expected values, whereas I thought of the expected value of the ratios. I guess I need more information about the data generation before moving on.
A: In cases where you cannot perform permutations, for example when the sample size creates millions of possibilities, another solution would be Monte Carlo resampling. 
The null hypothesis is that there is no difference in the speed
between $ext4$ and $xfs$, for $nocrypto$ and $crypto$. 
Therefore, the average ratio $\frac{ext4}{xfs}$ of all $nocrypto$ samples does not differ from that of $crypto$.
$H_{0}:T_{observed}=\frac{\sum x_{nocrypto} }{n_{nocrypto}}-\frac{\sum x_{crypto} }{n_{crypto}}=0
$
where
$
x=\frac{ext4}{xfs}
$
and
$
n=sample\, size
$
If $H_{0}$ is true, randomly picking results for ratios of  $nocrypto$ or $crypto$ would also result in $T_{observed}=0$. One would calculate:
$
T_{resampling}=\frac{x_{1}^{random}{+ 
x}_{n}^{random}}{n_{nocrypto}}-\frac{x_{1}^{random}{+ 
x}_{n}^{random}}{n_{crypto}}
$
and perform, say, 10,000 rounds of resampling. The resulting distribution of 
$T_{resampling}$ values is the confidence interval for $H_{0}$. The 
difference between $nocrypto$ and $crypto$ ratio is significant if the calculated $T_{observed}$ value lies outside the range of, e.g., 95% $(p < 0.05)$ of the $T_{resampling}$ values.
