When to use train test split? I am new to Machine Learning. I am trying to predict water levels and was given two data sets. 1 file is a Training data set with my water levels and factors that cause it, the other file is Testing data with only the factors and not including the actual water levels. I am thinking of using the method of Decisions trees, would I just make a model using my Training and then compare it to my Testing data?
p.s sorry if this sounds confusing
 A: As Henry says in a comment, this seems to be an exercise in which you will build a model on a specified training data set, then make predictions on another, test, data set. The actual outcomes (water levels) in the test set are hidden from you and will be used by someone else to evaluate the quality of the model you developed on the training data set.
From that perspective, you have the right idea about how to proceed. In addition to the reference suggested by Sycorax, An Introduction to Statistical Learning is a helpful guide to different modeling approaches.
To answer the question posed in the title, a strict train/test split like this is not usually reliable in practice unless you have tens of thousands of cases, as usεr11852 indicates in a comment. Frank Harrell discusses that in a blog post. With completely separate train and test sets, you tend to lose precision in developing the model and to lose power in evaluating model performance.
A better approach with smaller data sets is to build the model on the full data set and then check the performance of the modeling approach with resampling from the full data set and re-modeling the same way on each resample.
Bootstrapping is an efficient way to do that; you check models based on each of multiple bootstrap samples on the full data set.
Repeated cross validation is another, which fits into the general idea of train/test splits. You split the sample into, say, 10 subsets. You set aside one subset as a test set, build the model on the other 9 subsets, then test on the 10th subset. Repeat by holding out each of the subsets, remodeling, and re-testing. Then repeat the entire process many times on new reshuffled subsets.
A: You should always be using a train-test split, at a minimum (cross-validation being an extra step), whenever you are building a machine learning model. Splitting your data into a training set and a test set is necessary for model validation, because you want to be able to develop and optimize the model (using the training data) before testing its performance on previously unseen data (the test data).
The training set will typically be the majority of your data (a split of 80/20 or 70/30 is relatively common, but it really depends on the context of the question you are trying to answer), and you will use this data for the bulk of your workflow (EDA, feature engineering, model selection, hyperparameter tuning etc.). The test set is then held back as a means for testing the performance of your final model(s), intended to replicate out-of-sample prediction, to see if the model is generalizable and whether its performance will be sufficient in production.
So in your case, if you’re building a decision tree model, you would develop this model on the training set before testing the performance of the decision tree model on your test set. If the test set doesn't have the outcome in it, then I am guessing you are supposed to submit your model's predictions to someone/somewhere in order to assess performance?
