0
$\begingroup$

I have data with 95 numeric variables and 5 categorical variables. My Y has 2 values. I built a decision tree and my accuracy was 81.8%. Then I created 3 new variables as follows. It improved accuracy to 84.3%

  1. Normalize numeric variables and for training data, find mean vector for Y=1 and Y=0
  2. for each data point, find euclidean distance from each mean vector - distance0 and distance1
  3. third variable will be 0 if distance0 is <= distance1

I was wondering if there is any other new variables that i could create to improve the accuracy

I used a decision tree as it is fast to build and gives me indication whether a newly created variable is useful or not.

Please let me know if you have any thoughts

$\endgroup$

1 Answer 1

1
$\begingroup$

I was wondering if there is any other new variables that i could create to improve the accuracy

I used a decision tree as it is fast to build and gives me indication whether a newly created variable is useful or not."

Instead of just considering adding features, you can also consider removing features as decision trees are very prone to overfitting -- it would be helpful to know the gap between training and validation accuracy.

For removing features, you can try sequential feature selection. This is very simple to implement, and one off-the-shelf implementation for Python can be found in mlextend package: http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/

I am not sure how many data points you have, but random forests are usually better with dealing with larger number of features in terms of preventing overfitting (the main issue with using a decision tree classifier)

$\endgroup$
2
  • $\begingroup$ i have 40000 in training and 10000 test (i dont know their y value). I am using decision trees for preliminary model building. My final model would be either a neural network or a random forest $\endgroup$ Mar 19, 2019 at 3:19
  • $\begingroup$ for neural nets, manual feature engineering is usually not recommended as there is no benefit to it. The whole point behind deep learning is to automatic feature extraction -- let the neural net figure out how to combine features rather than doing it manually. To help with overfitting, try L2 regularization and/or dropout. My bet is that while you may improve the decision tree you may not ge the same benefits for the other two models. Just try it out using your engineered feature sets on those and compare. Also, like I suggested check for overfitting in the decision tree. $\endgroup$
    – resnet
    Mar 19, 2019 at 3:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.