I am about to begin running a data science competition. I have built a "baseline" model to test how accurate some of the competitors' submissions may be (I've set the target as to what they should aim to beat), and a simple, out-of-the-box decision tree model (
sklearn.tree.DecisionTreeClassifier) scored 100% accuracy, even after doing a train-test split.
This is a problem because there's no point in running a full DS challenge if some Average Joe off the street can just fit some black-box to the problem and get 100% accuracy. There needs to be room for error. I'm thinking of adding noise to the dataset (both the dependent and independent variables) to try to throw people off a bit to prevent overfitting. In what other ways, outside of adding noise and creating train and withheld-test datasets, can I modify the data to make this problem less "easy"?