3
$\begingroup$

I was reading the following blog post : http://epchan.blogspot.in/2013/01/the-pseudo-science-of-hypothesis-testing.html

Basically the author is discussing about using hypothesis testing to evaluate trading strategies, and writes:

You know the drill: the researchers first come up with a supposedly excellent strategy. In a display of false modesty, they then suggest that perhaps a null hypothesis can produce the same good return $R$. The null hypothesis may be constructed by running the original strategy through some random simulated historical data, or by randomizing the trade entry dates. The researchers then proceed to show that such random constructions are highly unlikely to generate a return equal to or better than $R$. Thus the null hypothesis is rejected, and thereby impressing you that the strategy is somehow sound.

1) If a person is an American then it is highly unlikely she is a member of Congress.

2) The person is a member of Congress.

3) Therefore it is highly unlikely she is an American.

The absurdity of hypothesis testing should be clear. In mathematical terms, the probability we are really interested in is the conditional probability that the null hypothesis is true given an observed high return $R$: $P(H_0|R$). But instead, the hypothesis test merely gives us the conditional probability of a return $R$ given that the null hypothesis is true: $P(R|H_0)$. These two conditional probabilities are seldom equal.

Now on Wikipedia, the general form of a statistical syllogism is given as:

1) A large proportion of $F$ are $G$

2) $I$ is an $F$

3) $I$ is a $G$

The author's example can be written as (provided Americans are $F$, Congress members are $G$ and the person is $I$):

1) A large proportion of $F$ are $\neg G$

2) $I$ is $G$

3) $I$ is $\neg F$

If the above syllogism is absurd, then that means "A large proportion of $F$ are $\neg G$" cannot imply that "A large proportion of $G$ are $\neg F$". Are there any special circumstances in which the former would imply the latter? Also, is the following syllogism correct?

1) A large proportion of $\neg F$ are $G$

2) $I$ is $G$

3) $I$ is $\neg F$

This would allow the conclusion that if a strategy A produces the return $R$, and if most of the good strategies produce the return $R$, then A is likely a good strategy.

$\endgroup$
  • 3
    $\begingroup$ This blog attacks a straw man by equating the p-value with the probability of the hypothesis. There are thoughtful, good assessments of the weaknesses of hypothesis testing, but evidently this isn't one of them. $\endgroup$ – whuber Apr 22 '17 at 13:41
  • 1
    $\begingroup$ Well the title of the blog article is certainly misleading, because I feel the author is only assessing a particular faulty application of hypothesis testing, rather than hypothesis testing in general. $\endgroup$ – u23 Apr 22 '17 at 13:54
  • $\begingroup$ It is true that null hypothesis statistical testing (NHST) has a weird logic in that it looks at P(R|H0) instead of a more intuitive P(H0|R) or P(H1|R). But there are practical and historical reasons why we frame NHST tests this way... I think there are much better critiques of NHST than this article. And any reader is free to use Bayesian analysis rather than NHTS... BTW, in retrospect, it's funny that he's using the accuracy of Nate Silver's predictions as evidence, since Nate Silver was as wrong as humanly possible in predicting the results of the 2016 U.S. presidential election. $\endgroup$ – Sal Mangiafico Aug 27 '17 at 18:56
2
$\begingroup$

Unfortunately cannot comment, but what definitely is not correct is your last statement. Just think about it in terms of overlapping circles, one for $\neg F$, one for $G$. Most of $\neg F$ might lie in $G$, but that doesn't imply that $G$ lies mostly in $\neg F$, the proportion of $\neg F$ in $G$ can be infinitely small.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.