4
$\begingroup$

I have two unknown distributions $A$ and $B$. I drew samples from them and store them in array a and b, respectively. I want to show that $E[A]$ is likely less than $E[B]$ (forgive my informal language, my statistics is too rusty) with a confidence level of 95%.

For this, I'm testing two null hypotheses with t-tests using SciPy:

  • $H_0: E[A] = E[B]$. The p-value returned by scipy.stats.ttest_ind(a, b, equal_var=False, alternative='two-sided') is p1.
  • $H_0: E[A] > E[B]$. The p-value returned by scipy.stats.ttest_ind(a, b, equal_var=False, alternative='greater') is p2.

What conditions must p1 and p2 satisfy for me to make the aforementioned conclusion? I know they must be small enough for me to reject these null hypotheses, but I'm not sure what values exactly to use.

Also, is this method even correct in the first place? I believe I can't make the conclusion by showing I fail to reject $H_0: E[A] < E[B]$, but I'm not sure whether the reverse way I described above is sound, either.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ What is your alternate hypothesis corresponding to $H_0: E(A) > E(B)$? $\endgroup$
    – B.Liu
    Commented Mar 15, 2021 at 8:47
  • 1
    $\begingroup$ @B.Liu $E[A] \le E[B]$ $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 8:48
  • 1
    $\begingroup$ I assume that $E[A]$ is the population mean of A and $E[B]$ is the population mean of B. Further I assume that you want to use frequentist null hypothesis significance testing. In frequentist approaches, we assume that the population means are constant, so it does not make sense to say that $E[A]$ is less likely than $E[B]$. Did you mean to say that the one is smaller than the other? $\endgroup$
    – Brandmaier
    Commented Mar 15, 2021 at 9:27
  • 1
    $\begingroup$ @Brandmaier I actually said "likely less" (i.e., likely smaller), not "less likely". $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 9:28
  • 1
    $\begingroup$ @B.Liu Hmm, say we were able to reject both $H_0: E[A] = E[B]$ and $H_0: E[A] > E[B]$ with $p < 0.05$. I'm not looking to quantify anything, but can we informally say that there is strong evidence supporting the hypothesis that $E[A] < E[B]$? $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 10:22

1 Answer 1

Reset to default
4
$\begingroup$

You are overthinking the problem a bit.

Based on your description, you have two samples from populations $A$ and $B$ where you want to show $\mathbb{E}(A) < \mathbb{E}(B)$. Assuming you drew enough samples and the populations do not follow distributions that are extremely skewed, you plan to use a $t$-test which is practically applicable.

A standard (read: conventional) way to do so is to specify

$$ H_0: \mathbb{E}(A) = \mathbb{E}(B),\quad H_1: \mathbb{E}(A) < \mathbb{E}(B)$$

and run on your Python:

scipy.stats.ttest_ind(a, b, equal_var=False, alternative='less')

(Note equal_var=False is copied directly from the question - you should consider whether this is actually the case, as it changes the result a bit).

Why your proposals do not work

$H_0: \mathbb{E}(A) = \mathbb{E}(B)$. The p-value returned by scipy.stats.ttest_ind(a, b, equal_var=False, alternative='two-sided')

Here you are testing against a two-sided alternate hypothesis $H_a: \mathbb{E}(A) \neq \mathbb{E}(B)$. Rejecting the null hypothesis in this case technically tells you nothing on which expectation is greater than which.

Moreover, from a practical perspective the p-value you get from the same samples will be double than that you get if you use $H_1$ as your alternate hypothesis, as you are also required to consider the other extreme in this hypothesis.

$H_0: \mathbb{E}(A) > \mathbb{E}(B)$. The p-value returned by scipy.stats.ttest_ind(a, b, equal_var=False, alternative='greater')

Firstly, the Python code you've suggested does not match the null hypothesis you have written, but the following

$$H_0: \mathbb{E}(A) = \mathbb{E}(B),\quad H_b: \mathbb{E}(A) > \mathbb{E}(B)$$

This is confirmed on the documentation for scipy.stats.ttest_ind, "This is a two-sided test for the null hypothesis that 2 independent samples have identical average (expected) values." there is no guarantee that the result (i.e. p-value) will extrapolate to the null of $\mathbb{E}(A) > \mathbb{E}(B)$.

Secondly, for such composite null hypothesis there is quite a lot of work involved to integrate the test statistic, p-value, etc. for every possible value in the set. As you can probably imagine, this gets pretty messy quick and I would not recommend doing so given the existence an easy alternative.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Ah! I see I misunderstood the ttest_ind function's parameters. So is it correct that showing the p-value returned by scipy.stats.ttest_ind(a, b, equal_var=False, alternative='less') is small enough alone is sufficient to reject $H_0 : E[A] = E[B]$ and show there is evidence supporting $H_1 : E[A] < E[B]$? $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 10:31
  • 1
    $\begingroup$ Yes, given your careful choice of wording! $\endgroup$
    – B.Liu
    Commented Mar 15, 2021 at 10:33
  • 1
    $\begingroup$ Thank you. I'm a little surprised that the alternative hypothesis need not be the complement of the null hypothesis. I guess my statistics class was too long ago :P $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 10:37
  • 1
    $\begingroup$ I've been thinking about this and I'm still not sure why our $H_1$ and $H_0$ can be nonexhaustive. Wouldn't this only make sense if we limit the problem space so that $E[A] > E[B]$ physically cannot happen? $\endgroup$
    – Zizheng Tai
    Commented Mar 15, 2021 at 11:05
  • 1
    $\begingroup$ @ZizhengTai This is actually a really good question - maybe this question and answer would help? Make sure to also read the linked questions in that Q&A. $\endgroup$
    – B.Liu
    Commented Mar 15, 2021 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.