I made a realization recently that might make clear that my process for model training might not be precise.

I almost always perform a train/test split early in the modeling process but I've begun questioning this process recently.

I was working on this data for detecting heart disease (binary response of has or doesn't have the disease) and I noticed that for each training sample I was getting different results (e.g. balanced accuracy of 0.6 then balanced accuracy of 0.9) and made the realization that there was a lot of variability in accuracy based on the training data at hand.

I'm familiar with k-fold cross-validation but I was wondering what you guys do in your modeling process. Do you still perform a train/test split in cases where you have only ~300 observations? Or do you only look at k-fold cross-validation techniques so you can consider the full dataset?

  • $\begingroup$ What kind of models are you using? $\endgroup$ – dante Apr 26 at 19:34
  • $\begingroup$ Logistic regression for binary classification. $\endgroup$ – Nicklovn Apr 26 at 20:47
  • $\begingroup$ Perhaps since your data set is so small, it would be better to use leave-one-out cross validation (LOOCV)? Not sure if this would give you a better estimate of your accuracy, but at least you would get a consistent result. $\endgroup$ – dante Apr 26 at 21:34
  • $\begingroup$ Certainly. That seems like a valid option. I'll keep that in mind. Thank you! $\endgroup$ – Nicklovn Apr 27 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.