3
$\begingroup$

I have paired samples of size 1000 from two distributions. I would like to test a null hypothesis that the 2.5% expected shortfalls1 of the two distributions are equal. How can I do that?

(This is a special case of another problem that deals with overlapping observations and some additional complications.)

1 $q$% expected shortfall is (the negative of) the expected value of the observations belonging to the left tail that is cut off at the $q$% quantile level. Synonyms of expected shortfall are conditional value at risk (CVaR), average value at risk (AVaR), expected tail loss (ETL), and superquantile.

$\endgroup$
2
  • $\begingroup$ How large is the sample size that you have? Roughly 97.5% of the sample will not tell much about the 2.5% tail. So if the sample size is small then you would need to make some assumptions about the distribution, describe it in terms of some parameteric distribution, such that the rest of the (small) sample can be useful in the prediction as well. $\endgroup$ Commented Oct 20, 2021 at 13:56
  • $\begingroup$ @SextusEmpiricus, my sample size is 1000 (one thousand) observations. Ideally I would not make parametric assumptions, but I understand this may be hard to avoid in small samples. $\endgroup$ Commented Oct 20, 2021 at 14:23

1 Answer 1

0
$\begingroup$

If you can assume that the two distributions are roughly normal or lognormal, this would be my approach.

Assume that $\ln(X)$ and $\ln(Y)$ are binormally distributed with means $m,n$, correlation $r$, and standard deviations $s,t$. Then there are formulas $ES(2.5\%, m, s)$ and $ES(2.5\%, n, t)$ for the expected shortfalls.

Meanwhile let $M,N$ be variables for the sample means of $\ln(X)$ and $\ln(Y)$. Let $Means(m,n,r,s,t)$ be the distribution of $M$ and $N$ under the given parameters, which is also binormal. Let $M_0$, $N_0$ be the values of these variables in this dataset.

Similarly let $R,S,T$ be variables for the sample correlation and standard deviations of $\ln(X)$ and $\ln(Y)$. Let $Devs(m,n,r,s,t)$ be the distribution of $R$, $S$ and $T$ under the given parameters, which also has an explicit approximation (recently in papers by Joarder, apparently originally going back to a 1915 paper by Fisher). Let $R_0$, $S_0$ , $T_0$ be the values of these variables in this dataset.

Now let the test statistic be $f(M,N,S,T) = ES(2.5\%, M,S) - ES(2.5\%, N,T)$. We assume that estimates of $M$ and $N$ are roughly independent of estimates of $R$, $S$ and $T$. So a reasonable first null hypothesis is that $f$ is roughly normal with variance the sum of \begin{align} Var_{M,N} &= Var[f(M, N, S_0, T_0)\, |\ M,N \sim Means(M_0, N_0, R_0, S_0, T_0)]\\ Var_{R,S,T}&=Var[f(M_0, N_0, S, T)\, |\, R,S,T \sim Devs(M_0, N_0, R_0, S_0, T_0)] \end{align}

These variances can be calculated numerically using the formulas above, and the final test is a z-test for $f(M_0, N_0, S_0, T_0)$ under the distribution $\mathcal{N}(0,Var_{M,N}+Var_{R,S,T})$.

$\endgroup$
7
  • $\begingroup$ What is the intuition (or formal argument) for $f$ being roughly normal? $\endgroup$ Commented Oct 20, 2021 at 16:41
  • $\begingroup$ My intuition is that the difference between two similar distributions should be roughly normal, which is why I said that this a reasonable first null hypothesis, but I don't have a formal argument for it. If the normality is a big concern you can generate lots of $M,N,R,S,T$ from the distributions and just see where $f(M_0,N_0,S_0,T_0)$ fits in the quantiles for $f(M,N,S,T)$. $\endgroup$
    – user225256
    Commented Oct 20, 2021 at 16:57
  • $\begingroup$ I simulated the difference between a pair of ES values each estimated from a sample from the same $N(0,1)$ distribution. The distribution I got is indeed roughly normal. Anyway, I am not really comfortable with an assumption of bivariate normality (or lognormality) for the data. $\endgroup$ Commented Oct 20, 2021 at 17:20
  • $\begingroup$ What can you assume about the distribution of the data? If you generate it using a jumpy process, then the numerical instability in calculated VaRs and expected shortfalls probably makes the whole question intractable. $\endgroup$
    – user225256
    Commented Oct 20, 2021 at 17:57
  • $\begingroup$ Unlike the more complicated question in the linked thread, here I assume a fixed distribution for all of the 1000 data points. So I guess there is no jumpy process, just 1000 i.i.d. observations. $\endgroup$ Commented Oct 20, 2021 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.