3
$\begingroup$

Every resource I have found states that the sample variance of a normal random variable follows a $\chi^2$-distribution such that $(n-1)\dfrac{s^2}{\sigma^2} \sim \chi^2_{n-1}$.

I am confused about how to find the $P$-value for a two-tailed test when the $s \approx \sigma$. Suppose $s$ is very slightly less than $\sigma$. I would expect that the $P$-value of a two-tailed test could be found by multiplying the left-tail area by $2$. However, this produces a $P$-value that is greater than $1$.

Many technologies instead use the right-tail area (which is less than $0.5$) and multiply that by $2$, but this does not sit well with me because (a) the sample variance is less than the population variance, and (b) this makes the upper limit for a $P$-value strictly less than $1$.

Am I missing something here? Is there a good way to calculate the $P$-value for this type of test that does not run into these issues?

example distribution

Edit:

To possibly clarify: my issue is that, intuitively, it seems like $s = \sigma$ should be the least extreme result, but $P(s^2\rm{\ more\ extreme\ than\ }\sigma^2) \ne 1$ using this approach. It seems like the distribution is implying that the least extreme result is $s = \sigma\sqrt{\frac{\rm{Med}(\chi^{2}_{n-1})}{n-1}} < \sigma$.

$\endgroup$

1 Answer 1

4
$\begingroup$

Note that the distribution is skewed.

The mistake is to take the mean ($E(s^2) = \sigma^2$) as the dividing line. If you want equal probability in each tail - as your post suggests by the mentioning of doubling - then you would look to see which side of the median you're on, rather than the mean. [If the left tail area exceeds 0.5 you're in the right tail. Given the "doubling", the rest isn't something you can just choose to change because you feel like it]

The odd behaviour you encounter is a direct result of not looking at the relationship to the median.

Now you can choose a different* measure of "at least as extreme"** from the one that results in equal-tail-probabilities; an appropriate choice for that may give something near to what you expect in terms of how you consider which "tail" you're in when looking at $s$ vs $\sigma$ but you'll instantly lose "double the tail area".

* perhaps the next most common choice would set the heights of the density in each tail to be the same (which gives shortest CIs) -- but this will make the effect you complain about even stronger, since then it's the relationship to the mode that says which tail you're in.

** (see the definition of a p-value)


Example

Let us explore a definition of "more extreme" that would imply $\sigma$ as the dividing line for which tail you're in.

Consider a definition where we make more extreme be based off the ratio $s/\sigma$. We now have to figure out which values of $s>\sigma$ are equally extreme to values where $s<\sigma$.

We could, for example say that $s/\sigma$ for $s>\sigma$ corresponds to $\sigma/s$ for $s<\sigma$. This would, for example say that $s=2\sigma$ is exactly as extreme as $s=\frac12\sigma$. Equivalently, we may say that "extremeness" is based on $|\log(s/\sigma)|$.

In our example, let's compare that with two other definitions of "more extreme":

Imagine we have $n=6$, so we have $5$ df. The mean of the chi-squared distribution with $\nu=5$ is $5$, the mode is $3$ and the median is about $4.35146$.

Now imagine we observe $s=2.8$ ($s^2=7.84$) and we're testing $\sigma=3$ ($\sigma^2=9$). What's the two-tailed p-value?

Under the null, $E(s^2)=9$, the mode of the distribution of $s^2$ is $5.4$ and the median is about $7.8326$.

  1. "more extreme" means "smaller tail area" (perhaps the most common approach in elementary texts):

    $s^2=7.84$ is at the $50.056$ percentile of the distribution of $s^2$ under the null (we're to the right of where there's 50% in each tail), so smaller tail area is to its right (not its left). The area to the right is 0.49944 and so the two-tailed p-value is 0.99888

    density of nu s^2 / sigma^2 showing equal tail areas either side of the median

    (the gap between the left and right tails is slightly exaggerated here; they're closer together than they appear in the diagram)

  2. "more extreme" means "lower density"

    Since the highest density (the mode) is to the left of our sample vale, we're in the right tail, and again there's 0.49944 area to the right. The density at $s^2=7.84$ (above the mode) is the same as the density at $s^2\approx 3.5279$ which has a left tail area of $0.14534$ for a two-tailed p-value of $0.64478$

    density of nu s^2 / sigma^2 showing small left tail for equal density either side of the mode

  3. "more extreme" means "$\log(s^2/\sigma^2)$ is further from $0$"

    Here $\log(s^2/\sigma^2)$ is $-0.13799$ (i.e. we're in the left tail), and equally extreme the other side would be $9/7.84 = 1.14796$, and we want the area above 9/7.84*5 on a $\chi^2_5$. The two tail areas are now $0.50056$ and $0.33237$ giving a two-tailed p-value of $0.83293$

    density of nu s^2 / sigma^2 showing small right tail for equal absolute log-ratio

Note that in each case, if you're consistent in your calculations (by sticking to the definition of more extreme rather than swapping from one to the other), no p-value can exceed 1.

[You may like to investigate likelihood ratio tests and ponder what the correct approach would be for that test in this case, with a small sample such as my example. In very large samples it will make no difference.]

$\endgroup$
5
  • $\begingroup$ My issue is that, intuitively, it seems like $s = \sigma$ should be the least extreme result, but $P(s^2\rm{\ more\ extreme\ than\ }\sigma^2) \ne 1$ using this approach. It seems like the distribution is implying that the least extreme result is $s = \sigma\sqrt{\frac{\rm{Med}(\chi^{2}_{n-1})}{n-1}} < \sigma$. $\endgroup$ Apr 19, 2018 at 15:51
  • $\begingroup$ Your desire to double tail areas is what makes the median of the distribution of $s$ the least extreme result (you can verify this for yourself quite easily). Your feelings can't change very simple algebraic facts. If you adopt some other measure of more extreme than "smaller tail area" (which makes the median as the dividing line) you then must give up on doubling tail areas and adopt a different approach to calculating two tailed p-values. So: which of the two mutually exclusive options do you want to give up - ... $\endgroup$
    – Glen_b
    Apr 20, 2018 at 0:18
  • $\begingroup$ (a) that the mean should be the boundary that determines which tail you're in or (b) doubling tail areas to get two tailed p-values? Simply pick one. If you want clarification of what I've said, please identify what you want explained. $\endgroup$
    – Glen_b
    Apr 20, 2018 at 0:18
  • 1
    $\begingroup$ I didn't comprehend why you seemed to simply be repeating the confusion in your question rather than addressing any of the points that I raised (in fact I was shocked by your response, since it seemed to treat the points in my answer as if they didn't exist at all). I apologize for any rudeness. I have added extensive example calculations. If you follow the same calculations and then consider trying to do these before there was wide availability of computers it may be easy enough to guess why people writing elementary books might tend to prefer the approach where you just double the p-value. $\endgroup$
    – Glen_b
    Apr 20, 2018 at 3:31
  • $\begingroup$ As for why it's still so widely used now even though people could do other things as easily, people in many application areas simply tend to teach what they were taught/do what "everybody else does". By contrast, a mathematical statistician might well lean toward using the "lower density is more extreme" option for this situation, which you might consider even more surprising than using the tail area, but has several good arguments for it. $\endgroup$
    – Glen_b
    Apr 20, 2018 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.