• One can safely see that under such stochasticity for the realizations of p-values and the distribution of its minimum, to get what people mean by 5% confidence (and the inferences they get from it), they need a p-value of at least one order of magnitude smaller.
On figure 2, there is a somewhat similar sounding claim, particularly the part "illusions of statistical significance":
Fig. 2. The probability distribution of a one-tailed p-value with expected value .11 generated by Monte Carlo (histogram) as well as analytically with ϕ(.) (the solid line). We draw all possible subsamples from an ensemble with given properties. The excessive skewness of the distribution makes the average value considerably higher than most observations, hence causing illusions of "statistical significance".
- What does Taleb mean by these statement?
- Is it correct that we should be using p-values order of magnitude lower to have "what people mean by 5% confidence"?