Think buying hundreds of fair dice. You do not know they are, though, and hence test if each has an expected value of 3.5 points, via throwing each many times (1000+). One of them must come up as "best", and if you do not account for multiple testing, almost certainly statistically significantly so.
Recall that the probability that a true null is rejected should (in practice, that may not be exactly true due to things like asymptotic approximations and finite-sample size distortions) not depend on sample size!
You might then conclude, wrongly (or at least not rightly, in that it is no better, but also no worse than the others), that this is the one you should bring to your next board game.
As to practical significance, this will indeed provide a clue, in that the "winning" one will likely have won with an average of points barely better than 3.5 when you tossed often.
Here is an illustration:
set.seed(1)
dice <- 100
throws <- 1000
tests <- apply(replicate(dice, sample(1:6, throws,
replace=T)), 2,
function(x) t.test(x, alternative="greater",
mu=3.5))
# right-tailed test, to look for "better" dice (assuming a
# game where many points are good, nothing hinges on this)
plot(1:dice, sort(unlist(lapply(1:dice, function(i)
tests[[i]]$p.value))))
abline(h=0.05, col="blue") # significance threshold not
# accounting for multiple testing
abline(h=0.05/dice, col="red") # Bonferroni threshold
max(unlist(lapply(1:dice, function(i) tests[[i]]$estimate)))
# the sample average of the "winner"

So we see a few "significantly" outperforming dice at level 0.05, but none, in this simulation run, after Bonferroni correction. The "winning" one (last line of the code) however has an average of 3.63, which is, in practice, not too far away from the true expectation 3.5.
We can also run a little Monte Carlo exercise - i.e., the above exercise many times so as to average out any "uncommon" samples that might arise from set.seed(1)
. We can then also illustrate the effect of varying the number of throws.
# Monte Carlo, with several runs of the experiment:
reps <- 500
mc.func.throws <- function(throws){
tests <- apply(replicate(dice, sample(1:6, throws,
replace=T)), 2,
function(x) t.test(x, alternative="greater",
mu=3.5))
winning.average <- max(unlist(lapply(1:dice, function(i)
tests[[i]]$estimate))) # the sample average of the "winner"
significant.pvalues <- mean(unlist(lapply(1:dice,
function(i) tests[[i]]$p.value)) < 0.05)
return(list(winning.average, significant.pvalues))
}
diff.throws <- function(throws){
mc.study <- replicate(reps, mc.func.throws(throws))
average.winning.average <- mean(unlist(mc.study[1,]))
mean.significant.results <- mean(unlist(mc.study[2,]))
return(list(average.winning.average,
mean.significant.results))
}
throws <- c(10, 50, 100, 500, 1000, 10000)
lapply(throws, diff.throws)
Result:
> unlist(lapply(mc.throws, `[[`, 1))
[1] 4.809200 4.108400 3.927120 3.692292 3.635224 3.542961
> unlist(lapply(mc.throws, `[[`, 2))
[1] 0.04992 0.05134 0.05012 0.04964 0.05006 0.05040
Hence, as predicted, the proportion of statistically significant results is independent of the number of throws (all proportions of $p$-values less than 0.05 are close to 0.05), while the practical significance - i.e., the distance of the average number of points of the "best" one to 3.5 - decreases in the number of throws.