I'm working on a project for uni where I have to predict a two-class problem, related to acceptance (or not) of a patent demand. Initially, I have a dataset separated into training and test data. My training data contains 259431 samples, while my test data contains 129715 samples, with 49 features (lots of which are categorical).
The professor gave us a small script to start treating the problem, and he's using a random forest, so I'm using that, at least for now.
Now, before directly sending the data to the Random Forest, I did some preprocessing on it, such as:
- Some variables were dates in the Month/Year format (such as 09/2004). So, I'm not sure if this is a good practice, but to give them a continuous sense, I converted them doing
Year*12 + Month.
- Next, I created dummy variables and replaced them with the original categorical variables. This step gave me a total of 474 features in my training data, and 442 features in my test data. This difference exists because, for the same categorical variable in the training and test, some categories do not appear in both.
- Also, I deliberately ignored some of the categorical variables because they were giving me too many different categories (which didn't seem relevant to me).
- What I did then was to check which features were present in the training and not in the test, and vice-versa, and used columns of 0s to include them. That gave me a total of 510 features.
- I then put the data through the
sklearn, to get rid of the
- Lastly, I scaled it all to a 0-1 range. This way the dummy variables won't be affected, and the other variables will be scaled. I'm not sure if this step is redundant for a random forest though, but decided to do it since it's not that much trouble anyway.
After all this, I put the data in a
n_estimators=500. The first time, I didn't define the
max_depth, but it took a lot of time to compute, so I used 5-fold cross validation around that range (5-30ish) and
max_depth=25 seemed to be a reasonable choice.
Finally, the scores for the training and test were calculated using
roc_auc_score. Which gives me around 0.96 for the training, and 0.60 for the test (rounded).
Now, here comes the part I'm wondering about:
We don't have direct access to the classes of the test data. They're available to the professor, and we only get the scores once we submit them via an online platform. We can also see a scoreboard, with the best scores of each student. And here, some people have managed to get up to 0.72 for the test data. So, I know for sure there's more I can do, but I'm having trouble figuring out what.
So, some questions:
- Does it make sense to do the preprocessing the way I did? Ignoring categorical variables with too many categories, and adding columns of 0s in the training and test to make them the same shape, for instance? Would it maybe be better to remove these features altogether instead of filling them with 0s?
- Also, we're not limited to using only one algorithm. So would it make sense to use something other than a random forest? Or maybe use an SVM in parallel and then average the results or something to that effect?
- The difference in scores (0.96 and 0.60) makes me wonder if the model is not overfitting. Not to mention, the cross validation scores were also around that range (0.60ish). Which seems awfully low to me. All in all, it feels like I need to do some feature selection before actually using the Random Forest, but I've also read that the Random Forest kinda computes the best features automatically, so would that end up being redundant?
Any other suggestions are welcome, and if you need any additional information, I'll try my best to put them here asap.