I made a realization recently that might make clear that my process for model training might not be precise.
I almost always perform a train/test split early in the modeling process but I've begun questioning this process recently.
I was working on this data for detecting heart disease (binary response of has or doesn't have the disease) and I noticed that for each training sample I was getting different results (e.g. balanced accuracy of 0.6 then balanced accuracy of 0.9) and made the realization that there was a lot of variability in accuracy based on the training data at hand.
I'm familiar with k-fold cross-validation but I was wondering what you guys do in your modeling process. Do you still perform a train/test split in cases where you have only ~300 observations? Or do you only look at k-fold cross-validation techniques so you can consider the full dataset?