Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories. I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue? Also, could you please share with me your thoughts about? Supposing a setup with same datasets, just different model structure where the stochasticity is purely given by intrinsic randomness of the optimization procedure (SGD). What I am asking is: 1) Why uncertainties are usually not quoted in association to ml scores? 2) If uncertainties are not quoted how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim? Let me propose a trivial example: I train model A on some data, and on a test set I get an f1 score of 80.0+-2.0, where I am quoting central value as the mean over N trainings and 2.0 is the standard deviation (assuming N is large enough). Then I train model B which is similar to model A but with a different topology (same dof as model A) and measure an f1 = 82.0+-(5.0). Would you claim model B is better than model A? Or would you consider the two scores to be statistically indistinguishable since they are compatible between each other in less then 1 sigma?

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories. I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue? Also, could you please share with me your thoughts about?

Supposing a setup with same datasets, just different model structure where the stochasticity is purely given by intrinsic randomness of the optimization procedure (SGD). What I am asking is:

1. Why uncertainties are usually not quoted in association to ml scores?
2. If uncertainties are not quoted how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim?

Let me propose a trivial example: I train model A on some data, and on a test set I get an f1 score of 80.0+-2.0, where I am quoting central value as the mean over N trainings and 2.0 is the standard deviation (assuming N is large enough). Then I train model B which is similar to model A but with a different topology (same dof as model A) and measure an f1 = 82.0+-(5.0). Would you claim model B is better than model A? Or would you consider the two scores to be statistically indistinguishable since they are compatible between each other in less then 1 sigma?

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories. I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue? Also, could you please share with me your thoughts about? While I agree with everything you mentioned this is not the point of my question. Supposing a setup with same datasets, just different model structure where the stochasticity is purely given by intrinsic randomness of the optimization procedure (SGD). What I am asking is: 1) Why uncertainties are usually not quoted in association to ml scores? 2) If uncertainties are not quoted how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim? Let me propose a trivial example: I train model A on some data, and on a test set I get an f1 score of 80.0+-2.0, where I am quoting central value as the mean over N trainings and 2.0 is the standard deviation (assuming N is large enough). Then I train model B which is similar to model A but with a different topology (same dof as model A) and measure an f1 = 82.0+-(5.0). Would you claim model B is better than model A? Or would you consider the two scores to be statistically indistinguishable since they are compatible between each other in less then 1 sigma?

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories. I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue? Also, could you please share with me your thoughts about? Supposing a setup with same datasets, just different model structure where the stochasticity is purely given by intrinsic randomness of the optimization procedure (SGD). What I am asking is: 1) Why uncertainties are usually not quoted in association to ml scores? 2) If uncertainties are not quoted how it is possible to compare different approaches and claim a possible improvement without a statistical confidence on the claim? Let me propose a trivial example: I train model A on some data, and on a test set I get an f1 score of 80.0+-2.0, where I am quoting central value as the mean over N trainings and 2.0 is the standard deviation (assuming N is large enough). Then I train model B which is similar to model A but with a different topology (same dof as model A) and measure an f1 = 82.0+-(5.0). Would you claim model B is better than model A? Or would you consider the two scores to be statistically indistinguishable since they are compatible between each other in less then 1 sigma?