Rajkomar, A., Oren, E., Chen, K., Dai, A.M., Hajaj, N., Liu, P.J., Liu, X., Sun, M., Sundberg, P., Yee, H. and Zhang, K., 2018. Scalable and accurate deep learning for electronic health records. arXiv preprint arXiv:1801.07860. https://arxiv.org/pdf/1801.07860.pdf

Model Evaluation and Statistical Analysis Patients were randomly split into development (80%), validation (10%) and test (10%) sets. Model accuracy is reported on the test set, and 1000 bootstrapped samples were used to calculate 95% confidence intervals. To prevent overfitting, the test set remained unused (and hidden) until final evaluation.

The study involved only 216,221 hospitalizations from 114,003 patients, so taking a single test set, especially with some highly imbalanced outcomes, seems to not make sense. Would not nested CV or bootstrapping the full set have been vastly superior, especially given that some of the diagnoses they were predicting (14,000 different ones) only had perhaps (guessing) 200 cases in the whole dataset ($p$ = 0.001). I think the test set is 20,000.

obs = []
for i in range(0,14000): obs.append(np.random.binomial(n=20000,p=0.001))

enter image description here

A hold out might have only caught 20 positive cases for some diseases (assuming most are rare). Further, bootstrapping the held out test set does not really help to detect this variation.

Generally, for their predictive models, I imagine Google uses a single hold out because their numbers are so large, but for these rare diseases this might be a real issue, and many diseases are rare.

Edit: published fast track https://www.nature.com/articles/s41746-018-0029-1

  • 2
    $\begingroup$ Arxiv isn't peer reviewed! $\endgroup$ – generic_user Feb 25 '18 at 22:17
  • $\begingroup$ @generic_user agree, but this paper has already gotten so much exposure $\endgroup$ – user0 Feb 25 '18 at 22:22
  • 1
    $\begingroup$ Is their dataset public? Try to overturn their result. Is it private? Nobody should trust it. $\endgroup$ – generic_user Feb 25 '18 at 22:45
  • 1
    $\begingroup$ I think the real tragedy is that they buried their (terrible) calibration results in the appendix. If I'm told at admission that my probability of dying is 50%, I would expect to die every other time, not 80% of the time... $\endgroup$ – M Turgeon Feb 28 '18 at 20:35
  • 1
    $\begingroup$ The dataset consists of medical records from patients who did not give explicit consent to this study. Although the records are "de-identified" and human-studies review boards at the hospitals allowed these investigators to perform these analyses, as provided in US regulations, those review boards might be hard pressed to allow public release of the data. The fear would be breaching patient confidentiality--might not such powerful machine-learning approaches, together with other publicly available information, somehow allow identification of the patients and thus their medical records? $\endgroup$ – EdM Feb 28 '18 at 21:33

I think that with over 100,000 patients, small sample sizes are not a consideration. Both traditional split-sample validation and complicated resampling techniques should lead to approximately similar model development and validation measures.

  • 2
    $\begingroup$ If the outcome is relatively common, then I agree. But what if it's an outcome with very low incidence? $\endgroup$ – user0 Mar 13 '18 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.