I would like to estimate extreme right tail quantiles (e.g. 99% or 99.5% tail) of a random variable $X$. The explicit representation of $X$ is unknown and there is no a-priori reason to assume it belongs to a well known parametric family. I do have access to simulated values of $X$ and I estimate my tail quantile by simulating a sample of a few thousand (maybe more) iid instances $(X_i)$ of $X$. Then I use the $\alpha N$ order statistic $T_{\alpha}[X] = X_{(\alpha N)}$ as an estimate of the tail quantile.
The problem is that I cannot simulate $X$ directly. I can only obtain noisy observations $(Y_i)$ of a distribution $Y= X + \epsilon$. The error $\epsilon$ has a normal distribution $\epsilon\sim \mathcal{N}(0, \sigma(x))$. While the size of $\sigma$ may be a function of $X$ the $\epsilon$ pertaining to different $X$ are independent. Since $X$ and thus also $Y$ depend on other risk factors in my simulation, I am able to obtain a few (say 3 to 5) repeated measurements of $Y(\omega)$ for the same underlying value $X(\omega)$ .
Sadly, the noise is very large in comparison to the size and differences of adjacent order statistics of $(Y_i)$. This leads to something very similar to the "regression to the mean" effect (see also here): My tail quantile estimate for $X$ obtained by the order statistic from the sample $(Y_i)$ is terribly biased upwards, i.e. much too large, due to the noise. This is the case even though I currently average over the repeated measurements, hence reducing the noise somewhat.
The next thing to try is to regress $X$ on $Y$, i.e. calculate the regression function $f(y)=E[X|Y=y]$. I am slightly reluctant, since this is a lot of work and hard to get right. I would like to make sure that there is no better alternative and I will obtain a sufficiently strong improvement of the estimate.
Questions
- What can be said about the size of the bias in the noisy estimate $T_{\alpha}[Y]$ versus the "clean" estimate value $T_{\alpha}[X]$?
- Are there general properties of $X$ (or ideally observable ones from $Y$) which drive or limit this error?
- Assume a good parameterisation of the regression function is possible and I find it. Does the regression approach lead to improved estimates of tail quantiles? Can this improvement be quantified? Practically speaking, at what value $y$ would I evaluate the regression function $f(y)$ to obtain an estimate for the $\alpha$ quantile of $X$?
- Are there other approaches to this estimation problem? Especially ones which do not require an explicit parameterisation of the regression function.
