I'm working with a tabular dataset with mostly dense features (around 40) and a few low cardinality (meaning around 10 possible values) categorical variables (around 20). In my experience, neural networks usually perform worse than gradient boosted trees when the dataset is tabular and most of the features are dense and the categorical features are low cardinality. My GBDT model gets a much better test performance than any NN I have trained on this dataset. It's probably also worth noting that I have a lot of data (millions of training examples).
The NN architecture I am using is a simple feedforward network.
What are some tricks I can try for matching the performance of GBDT models?
What I'm currently using
- Adam with initial learning rate selected by randomly sampling from a log scale
- Normalizing inputs to have mean 0, std 1
- Encoding categoricals as low dimensional embeddings and concatenating them to the dense features
- LayerNorm (BatchNorm does not improve training)
- 2 layers with 200 nodes (more layers seems more difficult to train. I have not tried more than 500 nodes per layer).
- Imputing missing values with the median
What I've tried but am not using
- Dropout (any amount of dropout causes the networks to underfit)
- Weight decay (similar to the dropout case, any amount causes the network to underfit)
- SGD (I didnt spend much time finding the best LR, but did not get as good of performance as Adam)
Things I've considered but decided not to try
- Using the leaf index from the GBDT as a high cardinality categorical feature and learning low dimensional representation in the NN
- Gradient clipping (my understanding is that this is most useful when training recurrent networks, but maybe I'm wrong)
- Feature interactions (Although I did try a vanilla factorization model)
- Learning rate schedules (I dont really have a reason for not trying this other than that there seem to be lots of options and no clear place to start)
Why I want a NN model
A NN model will allow us to use more complex loss functions. We're also interested in eventually bringing some high cardinality features. I expect that there is some trick I'm missing that is preventing us from getting similar performance to the GBDT models.