Nonparametric Confidence Intervals from Sample Quantiles - Sample Estimate Outside Upper and Lower Confidence Limits

Question

The Approach

Say I have a 95% confidence interval $(l, u)$ for some true parameter of interest $\theta$ computed from the 2.5th percent and 97.5th percent quantiles of a given set of continuously-distributed data. $\hat{\theta}$ falls within $(l, u)$. However, such an interval will not be symmetric around a given estimate $\hat{\theta}$.

I wish to now compute a confidence interval $(L, U)$ for some unknown (but presumably monotonic) function of $\theta$, $f(\theta)$, which is also of interest. This is straightforward (provided monotonicity holds):

$(L, U)$ = $\big(\frac{f({\hat{\theta})}}{u}, \frac{f({\hat{\theta})}}{l}\big)$.

Now, $f(\hat{\theta})$ should also lie within $(L, U)$. However, given my data (which is positive (having 1 as its lower endpoint), but negatively skewed), in certain cases, $f(\hat{\theta})$ lies entirely outside $(L, U)$.

Is there any reason for such ill behaviour? If so, what is the best way to get a well-behaved interval (if one in fact exists and is not given by $(1, \infty)$)?

My guess is that monotonicity does not in fact hold, or better yet, skewness present in the data is the culprit, thus rendering the interval essentially useless.

As a means to better address my question, and to aid other CV users who may benefit from such a question and any associated answers, I am providing a small example here.

The Data

My specific data is as follows:

   specs means       sds  
1      1 1.000 0.0000000
2      2 1.842 0.3651063
3      3 2.556 0.5545079
4      4 3.154 0.6951622
5      5 3.750 0.8129094
6      6 4.258 0.8791939
7      7 4.702 1.0233722
8      8 5.014 1.0138342
9      9 5.426 1.1040661
10    10 5.756 1.0500806
11    11 6.118 1.0818580
12    12 6.266 1.1034851
13    13 6.540 1.1184372
14    14 6.758 1.2290065
15    15 7.042 1.1396638
16    16 7.264 1.1971509
17    17 7.390 1.0845518
18    18 7.646 1.1416175
19    19 7.744 1.1157750
20    20 7.926 1.1433015
21    21 8.070 1.1540206
22    22 8.176 1.1487660
23    23 8.344 1.0732977
24    24 8.450 1.0629015
25    25 8.638 1.0024799
26    26 8.720 1.0544305
27    27 8.798 0.9995971   

The above data displays the sampling of 27 specimens (specs) along with the cumulative mean number of unique 'types' (means) one would expect to find from randomly sampling specs individuals.

Also included are the standard deviations (sds) associated with the mean number of unique types found for every specimen sampled. Here, the precise meaning of "types" is unimportant.

Sampling 1 specimen always results in finding exactly one "type" and the standard deviation associated with sampling the first specimen is always 0. The above data suggests we can expect to find 8.798 types on average from sampling 27 specimens at random, with a standard deviation of 0.9995971 types.

When the above data are plotted, a graph tending to a horizontal asymptote results. This asymptote corresponds to finding exactly 11 unique "types" (and always with a standard deviation of 0). I want to know how many specimens I likely need to sample to get all 11 "types"; that is, the number of specimens needed to "reach" the asymptote.

$f(\hat{\theta})$ is computed numerically via iterative resampling from an underlying probability distribution and this is how the curve is generated. These curves are generated for different species and show varying degrees of saturation from one species to the next. Thus, fitting a single parametric model to all species is not very realistic.

The above curve suggests that more sampling effort is needed for this species (since the slope of the curve near the endpoint is not close to 0). This makes sense since only 8.798 types out of the 11 total have been found.

Now, I can rely on the Central Limit Theorem to arrive at an approximate confidence interval via the Delta Method (using sds). However, this interval is symmetric and only valid with large sample sizes (*i.e., a high number of specs). Given the nature of the above graph, an asymmetric interval seems more plausible.

To do this, I compute the 2.5th (lower) and 97.5th (upper) percent quantiles for the cumulative mean number of unique "types"

lower    upper
    1    1
    1    2
    1    3
    2    4
    2    5
    2    6
    3    7
    3    7
    3    8
    4    8
    4    8
    4    8
    4    9
    4    9
    5    9
    5   10
    5    9
    6   10
    6   10
    6   10
    6   10
    6   10
    6   10
    6   10
    7   11
    7   11
    7   10

We see that sampling 27 specimens corresponds to finding between 7-10 types.

I wonder if I can use the 95% confidence interval $(l, u)$ = (7, 10) to get an interval for the number of specimens needed. Using the above approach (the function $f$ is not known), a reasonable interval is

$(L, U)$ = $(\frac{27*11}{10}, \frac{27*11}{7})$ = (29.7, 42.4)

However this interval excludes the value 27. I'm unsure as why this is the case and if there is a workaround.

Answer 1

You have some true interest-parameter $\theta$ and have computed a confidence interval for $\theta$, $(l,u)$. You also have an estimator of $\theta$, $\hat\theta$, which is between $l$ and $u$. That $(l,u)$ is a 95% CI means that $$ \DeclareMathOperator{\P}{\mathbb{P}} \P(l \le \theta \le u)=0.95 $$ I suppose the function $f$ is monotone increasing (it is in the example.) The monotone decreasing case is analogous. Then the above probability statement is equivalent to $$ \P(f(l) \le f(\theta) \le f(u))=0.95 $$ which means that a CI for $f(\theta)$ is $(f(l), f(u))$, and it is clear by monotonicity that this CI will contain $f(\theta)$.

Now there is a complication (see clarifications in comments under the Q.) The function $f$ is not exactly known, it is estimated by numerical simulation. If you can sample enough times to have a small monte carlo error, now the estimated CI $(\hat f(l), \hat f(u))$ where $\hat f$ is the estimated $f$ could be good enough as an approximation. If not, can we save the situation?

For the moment, I leave it here (it is late night.) Also, the situation is not completely clear to me ...