Your understanding of $p$-values seems to be correct.

Similar concerns are voiced quite often. What makes sense to compute in your example, is not only the number of studies out of 23 mln that arrive to false positives, but also the proportion of studies that obtained significant effect that were false. This is called "false discovery rate". It is not equal to $\alpha$ and depends on various other things such as e.g. the proportion of nulls across your 23 mln studies. This is of course impossible to know, but one can make guesses. Some people say that the false discovery rate is at least 30%.

See e.g. this recent discussion of a 2014 paper by David Colquhoun: Confusion with false discovery rate and multiple testing (on Colquhoun 2014). I have been arguing there against this "at least 30%" estimate, but I do agree that in some fields of research false discovery rate can be a lot bit higher than 5%. This is indeed worrisome.

I don't think that saying that null is almost never true helps here; Type S and Type M errors (as introduced by Andrew Gelman) are not much better than type I/II errors.

I think what it really means, is that one should never trust an isolated "significant" result.

This is even true in high energy physics with their super-stringent $\alpha\approx 10^{-7}$ criterion; we believe the discovery of the Higgs boson partially because it fits so well to the theory prediction. This is of course much much MUCH more so in some other disciplines with much lower conventional significance criteria ($\alpha=0.05$) and lack of very specific theoretical predictions.

Good studies, at least in my field, do not report an isolated $p<0.05$ result. Such a finding would need to be confirmed by another (at least partially independent) analysis, and by a couple of other independent experiments. If I look at the best studies in my field, I always see a whole bunch of experiments that together point at a particular result; their "cumulative" $p$-value (that is never explicitly computed) is very low.

To put it differently, I think that if a researcher gets some $p<0.05$ finding, it only means that he or she should go and investigate it further. It definitely does not mean that it should be regarded as "scientific truth".

Your understanding of $p$-values seems to be correct.

Similar concerns are voiced quite often. What makes sense to compute in your example, is not only the number of studies out of 23 mln that arrive to false positives, but also the proportion of studies that obtained significant effect that were false. This is called "false discovery rate". It is not equal to $\alpha$ and depends on various other things such as e.g. the proportion of nulls across your 23 mln studies. This is of course impossible to know, but one can make guesses. Some people say that the false discovery rate is at least 30%.

See e.g. this recent discussion of a 2014 paper by David Colquhoun: Confusion with false discovery rate and multiple testing (on Colquhoun 2014). I have been arguing there against this "at least 30%" estimate, but I do agree that in some fields of research false discovery rate can be a lot bit higher than 5%. This is indeed worrisome.

I don't think that saying that null is almost never true helps here; Type S and Type M errors (as introduced by Andrew Gelman) are not much better than type I/II errors.

I think what it really means, is that one should never trust an isolated "significant" result.

This is even true in high energy physics with their super-stringent $\alpha\approx 10^{-7}$ criterion; we believe the discovery of the Higgs boson partially because it fits so well to the theory prediction. This is of course much much MUCH more so in some other disciplines with much lower conventional significance criteria ($\alpha=0.05$) and lack of very specific theoretical predictions.

Good studies, at least in my field, do not report an isolated $p<0.05$ result. Such a finding would need to be confirmed by another (at least partially independent) analysis, and by a couple of other independent experiments. If I look at the best studies in my field, I always see a whole bunch of experiments that together point at a particular result; their "cumulative" $p$-value (that is never explicitly computed) is very low.

To put it differently, I think that if a researcher gets some $p<0.05$ finding, it only means that he or she should go and investigate it further. It definitely does not mean that it should be regarded as "scientific truth".

Your understanding of $p$-values seems to be correct.

Similar concerns are voiced quite often. What makes sense to compute in your example, is not only the number of studies out of 23 mln that arrive to false positives, but also the proportion of studies that obtained significant effect that were false. This is called "false discovery rate". It is not equal to $\alpha$ and depends on various other things such as e.g. the proportion of nulls across your 23 mln studies. This is of course impossible to know, but one can make guesses. Some people say that the false discovery rate is at least 30%.

See e.g. this recent discussion of a 2014 paper by David Colquhoun: Confusion with false discovery rate and multiple testing (on Colquhoun 2014). I have been arguing there against this "at least 30%" estimate, but I do agree that in some fields of research false discovery rate can be a lot bit higher than 5%. This is indeed worrisome.

I think what it really means, is that one should never trust an isolated "significant" result.

This is even true in high energy physics with their super-stringent $\alpha\approx 10^{-7}$ criterion; we believe the discovery of the Higgs boson partially because it fits so well to the theory prediction. This is of course much much MUCH more so in some other disciplines with much lower conventional significance criteria ($\alpha=0.05$) and lack of very specific theoretical predictions.

Good studies, at least in my field, do not report an isolated $p<0.05$ result. Such a finding would need to be confirmed by another (at least partially independent) analysis, and by a couple of other independent experiments. If I look at the best studies in my field, I always see a whole bunch of experiments that together point at a particular result; their "cumulative" $p$-value (that is never explicitly computed) is very low.

To put it differently, I think that if a researcher gets some $p<0.05$ finding, it only means that he or she should go and investigate it further. It definitely does not mean that it should be regarded as "scientific truth".

Your understanding of $p$-values seems to be correct.

Similar concerns are voiced quite often. What makes sense to compute in your example, is not only the number of studies out of 23 mln that arrive to false positives, but also the proportion of studies that obtained significant effect that were false. This is called "false discovery rate". It is not equal to $\alpha$ and depends on various other things such as e.g. the proportion of nulls across your 23 mln studies. This is of course impossible to know, but one can make guesses. Some people say that the false discovery rate is at least 30%.

See e.g. this recent discussion of a 2014 paper by David Colquhoun: Confusion with false discovery rate and multiple testing (on Colquhoun 2014). I have been arguing there against this "at least 30%" estimate, but I do agree that in some fields of research false discovery rate can be a lot bit higher than 5%. This is indeed worrisome.

I don't think that saying that null is almost never true helps here; Type S and Type M errors (as introduced by Andrew Gelman) are not much better than type I/II errors.

I think what it really means, is that one should never trust an isolated "significant" result.

This is even true in high energy physics with their super-stringent $\alpha\approx 10^{-7}$ criterion; we believe the discovery of the Higgs boson partially because it fits so well to the theory prediction. This is of course much much MUCH more so in some other disciplines with much lower conventional significance criteria ($\alpha=0.05$) and lack of very specific theoretical predictions.

Good studies, at least in my field, do not report an isolated $p<0.05$ result. Such a finding would need to be confirmed by another (at least partially independent) analysis, and by a couple of other independent experiments. If I look at the best studies in my field, I always see a whole bunch of experiments that together point at a particular result; their "cumulative" $p$-value (that is never explicitly computed) is very low.

To put it differently, I think that if a researcher gets some $p<0.05$ finding, it only means that he or she should go and investigate it further. It definitely does not mean that it should be regarded as "scientific truth".