Skip to main content
added 298 characters in body
Source Link
Dayne
  • 2.6k
  • 1
  • 9
  • 28

Let $X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Clearly, the two expressions are not same.

Now if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $p_xp_y$ as it is not uniformly distributed. It's cdf is $z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher typehigher type I error.

As such, I errordidn't need to assume anything about the distribution or test statistic. That was done just to illustrate that the two expressions are not the same.

The second part of the answer holds always because:

$$z-z\ln{z} > z \ \ \ \ \forall z \in [0,1)$$

Let $X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Now if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $p_xp_y$ as it is not uniformly distributed. It's cdf is $z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher type I error.

Let $X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Clearly, the two expressions are not same.

Now if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $p_xp_y$ as it is not uniformly distributed. It's cdf is $z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher type I error.

As such, I didn't need to assume anything about the distribution or test statistic. That was done just to illustrate that the two expressions are not the same.

The second part of the answer holds always because:

$$z-z\ln{z} > z \ \ \ \ \forall z \in [0,1)$$

added 298 characters in body
Source Link
Dayne
  • 2.6k
  • 1
  • 9
  • 28

Let $X \sim N(\mu,\sigma^2),$$X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

ThereforeNow,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

So the two expressions are not necessarily sameNow if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, it does leave open the possibility thatsame is not true for some family of distribution and some test statistic$p_xp_y$ as it may be possible to have the two expressions sameis not uniformly distributed. It's cdf is (it seems that$z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the cdf should be somenull hypothesis much more often leading to higher type of exponential functions)I error.

Let $X \sim N(\mu,\sigma^2),$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Therefore,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

So the two expressions are not necessarily same.

However, it does leave open the possibility that for some family of distribution and some test statistic it may be possible to have the two expressions same (it seems that the cdf should be some type of exponential functions).

Let $X \sim N(\mu,\sigma^2)$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Now,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

Now if I understand your question, you want to know what if we reject null based on $p_xp_y$ as compared to when we reject based on $p_{xy}$.

Under a true null hypothesis, p-value is uniformly distributed on $[0,1]$. So,

$$Pr(p_{xy} \leq 0.05) = 0.05$$

However, same is not true for $p_xp_y$ as it is not uniformly distributed. It's cdf is $z-z\ln{z}$. See this for derivation.

$$Pr(p_xp_y \leq 0.05) \approx 0.2$$

So you are clearly rejecting the null hypothesis much more often leading to higher type I error.

Source Link
Dayne
  • 2.6k
  • 1
  • 9
  • 28

Let $X \sim N(\mu,\sigma^2),$ and $\{x_i\}$ and $\{y_i\}$ be two same size i.i.d samples from $X$.

Now we do z-tests (assuming variance is known) individually for the two samples and then together. In both cases the null hypothesis is same:

$$H_0: \mu=0$$

Let $p_x$ and $p_y$ be respective p-values for individual tests and $p_{xy}$ be the p-value for combined test:

We know that under null hpothesis $$\bar{X}, \bar{Y} \sim N(0,\sigma^2/n)$$

Therefore,

$$p_x=Pr\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg| \geq Z \geq \bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg) = 2\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)$$ where $\Phi(.)$ is the cdf for $N(0,1)$

So, $$p_xp_y=4\Phi\bigg(-\bigg|\frac{\bar{x}}{\sigma/\sqrt{n}}\bigg|\bigg)\Phi\bigg(-\bigg|\frac{\bar{y}}{\sigma/\sqrt{n}}\bigg|\bigg)$$

whereas in the combined test:

$$p_{xy}=2\Phi\bigg(-\bigg|\frac{(\bar{x}+\bar{y})/2}{\sigma/\sqrt{2n}}\bigg|\bigg)$$

So the two expressions are not necessarily same.

However, it does leave open the possibility that for some family of distribution and some test statistic it may be possible to have the two expressions same (it seems that the cdf should be some type of exponential functions).