Problems with sampling with replacement for generating train, test, and validation data sets When creating train, test, and validation data sets for machine learning, random sampling without replacement is done to create disjoint data set partitions.  Is there anything wrong with using random sampling with replacement to generate train, test, and validation data sets.  Are there any advantages?
 A: What you want to do is to randomly partition the original data set. That means you randomly select a subset for training and use the rest for testing.
If there are the same data points in both training and test sets, then the performance of your ML algorithm will most likely be overestimated. 
For example, if you use random forest (RF) for classification, the data points from the training set would be perfectly 'predicted' in the test set. That would lead you to announce an inflated  performance of your solution and then get disappointed once the method is tried on the new data.
A: In some circumstances, many would recommend that you use random sampling with replacement instead of trying to generate separate train, validation, and test sets. Unless you have several thousands of cases, the hard 3-way split is going to limit your power because (1) you will not be using all the information in the data to develop the model, and (2) your final test set may have too few cases to give you a precise test of model performance. Furthermore, there's a danger that your apparent results will depend too much on the vagaries of the one specific data-set split that you performed to start.
A better approach to model development, particularly with less than several thousands of cases, can be to develop a model based on your entire data set (taking advantage of all the data that you have) and then to test the model-building performance with multiple (e.g., hundreds) of samples taken with replacement from your full original data set and of the same size at that data set: bootstrap samples. You repeat your model-building process on each of the bootstrapped samples, and then test the performance of each model on the full original data set. The idea is that the relation of each of the bootstrap samples to your original data set is similar to the relation of your original data set to the underlying population as a whole. So the average performance (e.g., in terms of bias) of the models based on the bootstrapped samples and applied to the whole data set provides an estimate of how well the model based on your whole data set would apply to the underlying population of interest. See this page and its links for more details. 
A: You want to randomly select a subset for training and a subset for  validation and a subset for testing.
If your dataset is small, then the p-value of your conclusions will be big. This is an unfortunate consequence of limited data and if you are reporting your p-values, then the decision makers based on your research know what is the risk they are accepting.
If you decide to take samples with replacement, then your testing and evaluation will be too optimistic because they are not independent of your training data.
