2
$\begingroup$

I am about to begin running a data science competition. I have built a "baseline" model to test how accurate some of the competitors' submissions may be (I've set the target as to what they should aim to beat), and a simple, out-of-the-box decision tree model (sklearn.tree.DecisionTreeClassifier) scored 100% accuracy, even after doing a train-test split.

This is a problem because there's no point in running a full DS challenge if some Average Joe off the street can just fit some black-box to the problem and get 100% accuracy. There needs to be room for error. I'm thinking of adding noise to the dataset (both the dependent and independent variables) to try to throw people off a bit to prevent overfitting. In what other ways, outside of adding noise and creating train and withheld-test datasets, can I modify the data to make this problem less "easy"?

$\endgroup$
6
  • 2
    $\begingroup$ Rather that modifying your existing data, would it be possible for you to switch to a more difficult problem/dataset? $\endgroup$
    – mkt
    Aug 2 '17 at 17:51
  • $\begingroup$ @mkt For the sake of this question, let's assume that I cannot switch to another dataset. The goal of the challenge is for competitors to flex their mathematical skills -- the problem they're solving isn't necessarily the focus here. I really want to showcase who's great at building mathematical models, not necessarily who's the best data scientist (which would incorporate SME knowledge). $\endgroup$
    – blacksite
    Aug 2 '17 at 18:27
  • 1
    $\begingroup$ Just to be clear here: This is 100% accuracy on a separate test dataset that would not be available to the participants and is separate from what the decision tree was trained on? If not: accuracy on the training dataset is not really a meaningful goal to look at. If it is on a spearate test dataset: besides picking a different example, one possibility might be to make the training dataset smaller. $\endgroup$
    – Björn
    Aug 2 '17 at 19:48
  • $\begingroup$ @Björn I found a slight error after I posted this; the 100% was on the training set... Bad form, I know. But I think my question still holds: in what other ways can I obfuscate or muddy the data a bit to make the classification task more difficult? $\endgroup$
    – blacksite
    Aug 2 '17 at 20:35
  • 1
    $\begingroup$ Where's the problem? 100% in the training set is not a problem. $\endgroup$
    – Björn
    Aug 3 '17 at 5:00
2
$\begingroup$

If you don't want to switch to an entirely different dataset (which to me seems to be the best way of solving this), you can do one or more of the following:

1. Change the target variable: Often a dataset contains more than one variable which makes sense to predict. Maybe there's a candidate which is more difficult to predict. Or you create a new target as some combination of existing variables (simple sums, linear combinations, whatever).

2. Delete variables: Look at which variables have the highest predictive power for your target and delete one or more of them.

3. Add variables: Pure noise variables are possible, but also transformed versions of existing variables. You can also combine one or more existing variables to form a new one. While this increases the likelihood that a very good model exists, it makes this good model more difficult to find.

$\endgroup$
1
$\begingroup$

Generally to avoid overfitting you can do regularization. So for a tree classifier you can limit the max_depth of trees,min_samples_leaf,min_leaf_nodes.You can also do pruning.Can you tell a bit about the problem as well? Is it a balanced or imbalanced dataset?

$\endgroup$
2
  • $\begingroup$ The dataset is balanced, yes (50/50). I can't limit the models a team wants to build. My only path of recourse is to make the outcome harder to predict by muddying the water. $\endgroup$
    – blacksite
    Aug 2 '17 at 19:48
  • 1
    $\begingroup$ Or is it possible to give them less number of features to build the model? What I mean to say is to remove some of the features from your dataset and let them try with the less number of features. $\endgroup$ Aug 2 '17 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.