Tricks for getting NNs to match the performance of GBDTs

Question

I'm working with a tabular dataset with mostly dense features (around 40) and a few low cardinality (meaning around 10 possible values) categorical variables (around 20). In my experience, neural networks usually perform worse than gradient boosted trees when the dataset is tabular and most of the features are dense and the categorical features are low cardinality. My GBDT model gets a much better test performance than any NN I have trained on this dataset. It's probably also worth noting that I have a lot of data (millions of training examples).

The NN architecture I am using is a simple feedforward network.

What are some tricks I can try for matching the performance of GBDT models?

What I'm currently using

• Adam with initial learning rate selected by randomly sampling from a log scale
• Normalizing inputs to have mean 0, std 1
• Encoding categoricals as low dimensional embeddings and concatenating them to the dense features
• LayerNorm (BatchNorm does not improve training)
• 2 layers with 200 nodes (more layers seems more difficult to train. I have not tried more than 500 nodes per layer).
• Imputing missing values with the median

What I've tried but am not using

• Dropout (any amount of dropout causes the networks to underfit)
• Weight decay (similar to the dropout case, any amount causes the network to underfit)
• SGD (I didnt spend much time finding the best LR, but did not get as good of performance as Adam)

Things I've considered but decided not to try

• Using the leaf index from the GBDT as a high cardinality categorical feature and learning low dimensional representation in the NN
• Gradient clipping (my understanding is that this is most useful when training recurrent networks, but maybe I'm wrong)
• Feature interactions (Although I did try a vanilla factorization model)
• Learning rate schedules (I dont really have a reason for not trying this other than that there seem to be lots of options and no clear place to start)

Why I want a NN model

A NN model will allow us to use more complex loss functions. We're also interested in eventually bringing some high cardinality features. I expect that there is some trick I'm missing that is preventing us from getting similar performance to the GBDT models.

Answer 1

A lot of the same techniques to training CNNs and other architectures also apply to training fully connected networks. A list of things that I tried and gave me much better performance:

• Used SGD with a carefully selected learning rate and learning rate schedule. Using nesterov momentum with a momentum of 0.9 will also likely speed convergence. Adaptive methods are easy to configure, but SGD will do better when properly tuned [1].
• Choose your learning rate based on a subset of the data with the batch size you want to use. SGD convergence is robust to the size of the dataset [5] section 1.
• Use a smaller batch size, but also increase the initial learning rate. Some works have suggested a linear scaling rule, while others have suggested a square root scaling [2,3]. Warm up the learning rate from a smaller value [2]
• If using LayerNorm, instead use a simple variant that does not learn bias and variance parameters (these tend to overfit and make the network more difficult to train)[4].
• If using BatchNorm, be aware of how it is affected by distributed training [2]
• Monitor network statistics every several iterations. Amount of activation saturation, gradient norms, ratio of the gradient norm to the weight norm and the weight norm can all be informative to how to further tune parameters.
• Choose layer-wise initial learning rates. This can be done either using second order information (the inverse of the maximum eigenvalue of the hessian) or just first order information (ensure the ratio of the gradient norm to the weight norm is some common value such as 0.1) [5] chapter 1 and 18.
• Local minimum in overparametrized networks are usually good enough [6].

Some personal anecdotal points:

• Somehow, deeper networks are easier to optimize than wider networks
• Bengio [5] section 19 claims that networks with fixed layer size work better than varying layer size. He also claims that using a wider layer size than the input dimension works better, although I have found that using a smaller size is easier to optimize.

References

[1] Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." Advances in neural information processing systems. 2017.

[2] Goyal, Priya, et al. "Accurate, large minibatch sgd: Training imagenet in 1 hour." arXiv preprint arXiv:1706.02677 (2017).

[3] Krizhevsky, Alex. "One weird trick for parallelizing convolutional neural networks." arXiv preprint arXiv:1404.5997 (2014).

[4] Xu, Jingjing, et al. "Understanding and Improving Layer Normalization." Advances in Neural Information Processing Systems. 2019.

[5] Orr, Genevieve B., and Klaus-Robert Müller, eds. Neural networks: tricks of the trade. Springer, 2003.

[6] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.