I am looking to train a machine learning model to solve a regression problem. The targets, y, are not normally distributed. It is perhaps best described as somewhat bimodal. For context, I have 13,000 points in total.
I am looking to have an 80%/20% train/test split, and on the training set, I intend to do 5-fold cross-validation. My concern is that, depending on the seed used to split the data, there can be somewhat significant run-to-run variability because the distribution is sampled randomly and the subsamples do not have a similar distribution as the parent dataset or between other subsamples.
For this reason, I am considering doing stratified sampling with the train/test split as well as during the 5-fold cross-validation. My plan is to set up five strata of the type (-$\infty$,1], (1,2], (3,4], (4,+$\infty$] and have the subsamples contain a similar distribution as the parent dataset.
However, I see very few people in my field do stratified sampling for regression problems and am concerned about unforeseen circumstances. What concerns are there that one should consider when it comes to stratified sampling in machine learning? Of course, I can also run the ML regression several times and average the results, regardless of whether I do stratified sampling or purely random sampling.