Return to Answer

All models are wrong, but some are useful. Expressions about significance are estimates and typically wrong, but often not so bad and therefore still useful. Recently, since more and more research is aiming to zoom in on small effects occurring in populations with large variation, being magnified by cranking up the sampling number, the previously useful models for estimating errors (assuming sampling error is dominating systematic error and the latter can be neglected) are becoming less accurate and less useful.

All models are wrong, but some are useful. Expressions about significance are estimates and typically wrong, but often not so bad and therefore still useful. They are not so bad because the assumption that sampling error is dominating systematic error often works and the latter can be neglected. However, recently, these (previously) useful models for estimating errors are becoming less correct and less useful. This is because more and more research is able to zoom in on small effects occurring in populations with large variations. The small effects are being magnified by cranking up the sampling size. But when we look at small effects and small sampling noise (due to large samples) then the systematic error can not be neglected anymore.

We do not want to move away from significance. Significance is important. It is an indicator that a dataset is large/significant enough in order for some observed effect to be unlikely due to random noise. We still want experimenters to aim for experiments that will be significant. Insignificant experiments, those which likely reflect noise, are not very useful; the interpretation of the outcome is uncertain (is it a 'true' effect or is it noise?). Significance means that the experiment is able to give outcomes with relatively more certain interpretations (the outcome is likely not noise but instead some true falsification of the null hypothesis).

That means that, eventhough significance means that it is something unlikely to occur given the present hypothesis predicting no effect, it is likely for a researcher to find a significant result while it isn't there.

This makes that we now have a big noodle-soup of reports on research data with only tiny effects. If something is a big effect then it is likely to have already been proven. But, we are now having an enormous army of eager (and pressurised) scientists trying to find something new, so they will focus on something (anything) small and by doing a significant experiment make it big.

A problem with significance is in the methodology to express errors occurring between experiments only based on the error occuring within an experiment.

The current experimental scientific 'world' is being driven by these incentives to publish significant (it doesn't matter what) rather than meaningful. The problem with that is that the due to technological developments we have been able to increase the scale of experimental work and do massive testing, allowing to make small effects significantly visible. This places a focus on finding small differences in parameters of the population distributions (it's resourceful niche for many researchers), while the individual people within those populations have much more variation and differences.

We have a focus on the average, rather than the specific/individual, because differences between averages, no matter how small, can be made be easily made significant (in practice not always easy, but the principle is simple, it is just increasing the quantity of testing).

If you measure tiny effects, and make them significant purely by increasing the sample number, then your are not anymore certain that the determined effect is due to a discrepancy in the null model, it can also be the sampling procedure (When a significance test fails we tend to say that the null hypothesis is falsified, but we should say that the null hypothesis plus the experiment is falsified. However we do not normally say that because for large enough effects we tend to ignore the systematic effects).

We do not want to move away from significance. Significance is important. It is an indicator that a data set is large/significant enough in order for some observed effect to be unlikely due to random noise. We still want experimenters to aim for experiments that will be significant. Insignificant experiments, those which likely reflect noise, are not very useful; the interpretation of the outcome is uncertain (is it a 'true' effect or is it noise?). Significance means that the experiment is able to give outcomes with relatively more certain interpretations (the outcome is likely not noise but instead some true falsification of the null hypothesis).

That means that, even though significance means that it is something unlikely to occur given the present hypothesis predicting no effect, it is likely for a researcher to find a significant result while it isn't there.

This makes that we now have a big noodle-soup of reports on research data with only tiny effects. If something is a big effect then it is likely to have already been proven. But, we are now having an enormous army of eager (and pressurized) scientists trying to find something new, so they will focus on something (anything) small and by doing a significant experiment make it big.

A problem with significance is in the methodology to express errors occurring between experiments only based on the error occurring within an experiment.

The current experimental scientific 'world' is being driven by these incentives to publish significant (it doesn't matter what) rather than meaningful. The problem with that is that due to technological developments we have been able to increase the scale of experimental work and do massive testing, allowing to make small effects significantly visible. This places a focus on finding small differences in parameters of the population distributions (it's resourceful niche for many researchers), while the individual people within those populations have much more variation and differences.

We have a focus on the average, rather than the specific/individual, because differences between averages, no matter how small, can easily be made significant (in practice not always easy, but the principle is simple, it is just increasing the quantity of testing).

If you measure tiny effects, and make them significant purely by increasing the sample size, then your are not anymore certain that the determined effect is due to a discrepancy in the null model, it can also be the sampling procedure (When a significance test fails we tend to say that the null hypothesis is falsified, but we should say that the null hypothesis plus the experiment is falsified. However we do not normally say that because for large enough effects we tend to ignore the systematic effects).

Significance is not really a tool in model selection. Significance is a tool in hypothesis testing and in verifying the (statistical) validity of conclusions that may stem from such test (a conclusion should, with reasonable probability, not be due to random noise).

With significance testing you often have a preference for the null hypothesis/model. The goal of the experiment is not model selection, but instead model rejection. Significance testing is done to trial/test whether the null hypothesis is correct (and often the test is made with an alternative hypothesis in hindsight such that the test has a high probability/power to reject the null hypothesis if the specific alternative is true).

In these kind of trials you do get the situation that there might be multiple models against which the null hypothesis can be tested and the idea might be to see which of these models make most sense. This does resemble a lot model selection and the concepts can performed in a mixed way, but from my point of view they should not be considered mixed. E.g. one may test multiple factors and see whether any of them has a significant effect. You could see this as model selection, seeing which factor is the best model... However, it is in principle more like performing multiple null hypothesis tests (each hypothesis being that a specific factor has no effect).

Model selection is an optimization which can be worked out without significance (if you have an appropriate loss function). If your are doing some optimization, e.g. predicting, then bootstrapping might indeed be a good way to not only test the variance of the estimates, but also the bias.