How to reproduce the distribution of p-values in a Monte Carlo? In whichever program (R preferred, but pseudo-code would do), could I get an idea of how Nassim Taleb simulated the distribution of p-values - I guess under the alternative hypothesis - on this MOOC?
Somehow he has to draw from two identical distributions that are offset in mean and or variance, in such a way that the alternative hypothesis is true, but he also knows the "true p-value" as $11$ in his simulation, and I wonder if this is the mean that he got after a random simulation of two Gaussians with different means, or whether he started with this $p<0.11$ value to retrofit the parameters of his distributions.

Without prefixing a p-value as a desired mean, it is quite easy:
n <- 10000
sam <- 1000
p.vals <- vector(,n)
for(i in 1:n){
u <- rnorm(sam,0.1,1)
v <- rnorm(sam,0,1)
p.vals[i] <- t.test(u,v)$p.value
}
hist(p.vals, border=F)
abline(v=mean(p.vals))
legend(mean(p.vals),3000,paste('Mean of p-values is', round(mean(p.vals),2)))


 A: This is more of a comment really - There are some parts of Taleb's lecture that either I don't understand or I don't agree with.
Taleb simulates the same experiment several times and he averages the p-values. In his setup the simulation is such that the average of the p-values is 0.11. His point is that because most of the p-values are well below the average, the p-value is a deceiving statistic. This is shown nicely in your histogram.
My objection is that averaging p-values doesn't make sense. I can't tell why formally but I don't think probabilities is something you can average straightaway. Instead I would take the mean of the log-transformed pvalues:
set.seed(1234)
n <- 10000
sam <- 1000
p.vals <- vector(,n)
for(i in 1:n){
    u <- rnorm(sam,0.1,1)
    v <- rnorm(sam,0,1)
    p.vals[i] <- t.test(u,v)$p.value
}
logp <- -log(p.vals)
hist(logp, border= 'white', col= 'grey60', xlab= sprintf('-log(p-value)'), main= '')
abline(v=mean(logp))

mean(p.vals)
[1] 0.1085658

exp(-mean(logp))
[1] 0.0167




The anti-log of this mean in this case is ~0.017, much smaller than 0.11. In my opinion, 0.017 is closer to "gut feeling" since you are simulating samples of 2000 data points, quite large to detect a difference of 0.1 with sd= 1.
Other loosely related objections I have: Taleb says that p-values are stochastic - I see that since they are computed from random samples. However, the fact that a metric is stochastic doesn't mean that it is necessarily biased, which I think is what he let the audience understand. Also, the fact that p-values are abused and misunderstood (I agree), doesn't make p-values wrong or deceiving. Testing the same hypothesis multiple times will give small a pvalue in the long run and this is how p-values work, I see nothing wrong with it.
