Methods of preventing overfitting other than adding noise to data? I am about to begin running a data science competition. I have built a "baseline" model to test how accurate some of the competitors' submissions may be (I've set the target as to what they should aim to beat), and a simple, out-of-the-box decision tree model (sklearn.tree.DecisionTreeClassifier) scored 100% accuracy, even after doing a train-test split.
This is a problem because there's no point in running a full DS challenge if some Average Joe off the street can just fit some black-box to the problem and get 100% accuracy. There needs to be room for error. I'm thinking of adding noise to the dataset (both the dependent and independent variables) to try to throw people off a bit to prevent overfitting. In what other ways, outside of adding noise and creating train and withheld-test datasets, can I modify the data to make this problem less "easy"?
 A: If you don't want to switch to an entirely different dataset (which to me seems to be the best way of solving this), you can do one or more of the following:
1. Change the target variable: Often a dataset contains more than one variable which makes sense to predict. Maybe there's a candidate which is more difficult to predict. Or you create a new target as some combination of existing variables (simple sums, linear combinations, whatever).
2. Delete variables: Look at which variables have the highest predictive power for your target and delete one or more of them.
3. Add variables: Pure noise variables are possible, but also transformed versions of existing variables. You can also combine one or more existing variables to form a new one. While this increases the likelihood that a very good model exists, it makes this good model more difficult to find.
A: Generally to avoid overfitting you can do regularization. So for a tree classifier you can limit the max_depth of trees,min_samples_leaf,min_leaf_nodes.You can also do pruning.Can you tell a bit about the problem as well? Is it a balanced or imbalanced dataset?
