3
$\begingroup$

An acquaintance of mine has been using this wrong inference formula for years: given

  • a i.i.d. sample $\mathbf{X}={X_1,\dots,X_N}$ for a continuous RV $X$,
  • sample mean $\bar{X}=\frac{\sum X_i}{N}$ and sample standard deviation $\bar{\sigma}=\frac{\sum \left(X_i-\bar{X}\right)^2}{N-1}$

estimate the 0.95-quantile $q_{0.95}$ as

$$q_{0.95} = \bar{X} + 2 \bar{\sigma}$$

(which is not even a decent point estimate - you should at the very least use $q_{0.95} = \bar{X} + 1.645\bar{\sigma}$).

What are the correct confidence intervals for a generic $q$, in the three cases:

  • we know nothing on $X$ (apart from the fact that it's continuous), and we look for an exact (non-asymptotic) answer. I think this answer for the median could be modified for a generic quantile
  • as before, but an asymptotic solution is fine. I guess there should be at least a couple answers here...one for quantiles which aren't close to 0 or 1, and one for quantiles which are. Maybe one based on normal approximation and one based on Poisson?
  • finally, we assume $X$ to have a Gaussian distribution with unknown mean and variance.
$\endgroup$
14
  • 3
    $\begingroup$ Your colleague’s formula gives just a point estimate, not an interval. Which are you after? That formula also isn’t necessarily an awful estimator of $q_{0.975}$. I might run some simulations to check out its MSE compared to other estimates. My guess is that it’s pretty good for a normal distribution and then fails for asymmetric and heavy-tailed distributions. $\endgroup$
    – Dave
    Feb 9 '20 at 12:37
  • 1
    $\begingroup$ I'll offer my response to this Cross Validated question, which discusses two approaches for a confidence intervals for a quantile: 1) an approach from Conover based on the binomial distribution; and, 2) confidence intervals by percentile bootstrap. $\endgroup$ Feb 9 '20 at 14:21
  • 2
    $\begingroup$ Confidence intervals for quantiles are commonly known as tolerance intervals. $\endgroup$
    – whuber
    Feb 9 '20 at 15:31
  • 1
    $\begingroup$ I think that is incorrect. Wouldn’t you want to work with $q_{0.95}$ instead of $0.975$? $\endgroup$
    – Dave
    Feb 9 '20 at 18:20
  • 1
    $\begingroup$ Yes, the idea looks right. I cannot discern any general relationship between the original $0.975$ value and the $95\%,95\%$ criteria, though. Don't you mean "$0.95$"? $\endgroup$
    – whuber
    Feb 9 '20 at 18:20
3
$\begingroup$
  • The first case was answered in detail in this question.
  • One example of the second case is shown here, where the authors apply a normal approximation to the binomial distribution used in calculations of the first case.

The third case is given by Hahn and Meeker in their handbook Statistical Intervals (2nd ed., Wiley 2017):

A two-sided $100(1-\alpha)\%$ confidence interval for $x_q$, the $q$ quantile of the normal distribution, is

$$ \left[\bar{x}-t_{(1-\alpha/2;\,n-1,\,\delta)}\frac{s}{\sqrt{n}},\;\bar{x}-t_{(\alpha/2;\,n-1,\,\delta)}\frac{s}{\sqrt{n}}\right] $$ where $t_{(\gamma;\,n-1,\,\delta)}$ is the $\gamma$ quantile of a noncentral $t$-distribution with $n-1$ degrees of freedom and noncentrality parameter $\delta = -\sqrt{n}z_{(q)}=\sqrt{n}z_{(1-p)}$.

Here, $z_{(q)}$ denotes $\Phi^{-1}(q)$, the $q$ quantile of the standard normal distribution.

For example, let's assume we drew $n=20$ samples from a normal distribution with unknown mean and standard deviation. The sample mean was $\bar{x}=10.5$ and the sample standard deviation was $s=3.19$. Then, the two-sided $95\%$ confidence interval for the $q=0.25$ quantile $x_{0.25}$ would be given by $(6.42; 9.76)$.

Here is some R code and a small simulation to check the coverage. By changing the parameters, you can run your own simulations:

normquantCI <- function(x, conf_level = 0.95, q = 0.5) {
  
  x <- na.omit(x)
  n <- length(x)
  xbar <- mean(x)
  s <- sd(x)
  ncp <- -sqrt(n)*qnorm(q)
  tval <- qt(c((1 + conf_level)/2, (1 - conf_level)/2), n - 1, ncp)
  se <- s/sqrt(n)
  
  xbar - tval*se
  
}

# Simulate the coverage

set.seed(142857)

q <- 0.25 # Quantile to calculate the CI for
conf_level <- 0.95 # Confidence level
true_mean <- 100 # The true mean of the normal distribution
true_sd <- 15 # True sd of the normal distribution
sampsi <- 20 # The sample size

trueq <- qnorm(q, true_mean, true_sd) # The true quantile

res <- replicate(1e5, {
  citmp <- normquantCI(rnorm(sampsi, true_mean, true_sd), conf_level = conf_level, q = q)
  ifelse(citmp[1] < trueq & citmp[2] > trueq, 1, 0)
})

sum(res)/length(res)
[1] 0.95043
$\endgroup$
3
  • $\begingroup$ This is nice and I upvoted (btw, made a minor edit, please check it out). However, I don't think you need to link to an external site for the sake of answering point 2 (the asymptotic case)....I can copy that formula into your answer if you don't feel like doing it yourself. It would also be nice to include the other asymptotic for the other case (when $n$ is large but $p$ is either very close to 0 or 1). I think a Poisson approximation would work, even though probably extreme events theory is the right way to go? $\endgroup$
    – DeltaIV
    Feb 9 '20 at 16:31
  • $\begingroup$ @DeltaIV Thanks. I deliberately didn't copy the formula for the normal approximation because I wanted to focus on the third case. I suspect that the formula wouldn't make much sense without more context from the first case. Instead of copy & pasting the formula into my answer, I would encourage you to post it as a separate answer, preferably with a bit more context. You could also develop and present your ideas about the Poisson approximation there. $\endgroup$ Feb 9 '20 at 16:42
  • $\begingroup$ Rather than having three answers, each focusing on one point, I think it would be better to have one answer covering all three points and ideally comparing them (it would also make it easier for me to accept an answer). As a matter of fact, initially the question title was "comparison among confidence [..]", but then it took the "comparison" part away, because I worried that people may choose not to answer, because they wouldn't want to spend time writing a lot of R code. Anyway, if you prefer to focus on the third point only, I'll do as you say & write a separate answer. $\endgroup$
    – DeltaIV
    Feb 9 '20 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.