I'm working with a tabular dataset with mostly dense features (around 40) and a few low cardinality (meaning around 10 possible values) categorical variables (around 20). In my experience, neural networks usually perform worse than gradient boosted trees when the dataset is tabular and most of the features are dense and the categorical features are low cardinality. My GBDT model gets a much better test performance than any NN I have trained on this dataset. It's probably also worth noting that I have a lot of data (millions of training examples).

The NN architecture I am using is a simple feedforward network.

What are some tricks I can try for matching the performance of GBDT models?

What I'm currently using

  • Adam with initial learning rate selected by randomly sampling from a log scale
  • Normalizing inputs to have mean 0, std 1
  • Encoding categoricals as low dimensional embeddings and concatenating them to the dense features
  • LayerNorm (BatchNorm does not improve training)
  • 2 layers with 200 nodes (more layers seems more difficult to train. I have not tried more than 500 nodes per layer).
  • Imputing missing values with the median

What I've tried but am not using

  • Dropout (any amount of dropout causes the networks to underfit)
  • Weight decay (similar to the dropout case, any amount causes the network to underfit)
  • SGD (I didnt spend much time finding the best LR, but did not get as good of performance as Adam)

Things I've considered but decided not to try

  • Using the leaf index from the GBDT as a high cardinality categorical feature and learning low dimensional representation in the NN
  • Gradient clipping (my understanding is that this is most useful when training recurrent networks, but maybe I'm wrong)
  • Feature interactions (Although I did try a vanilla factorization model)
  • Learning rate schedules (I dont really have a reason for not trying this other than that there seem to be lots of options and no clear place to start)

Why I want a NN model

A NN model will allow us to use more complex loss functions. We're also interested in eventually bringing some high cardinality features. I expect that there is some trick I'm missing that is preventing us from getting similar performance to the GBDT models.

  • $\begingroup$ +1 because you obviously invested some time on this. That said, GBDT's are not simple classifiers/regressors so recreating through an NN is not trivial or even proven to be possible. What would be a loss function you would want minimised? $\endgroup$
    – usεr11852
    Commented Aug 24, 2020 at 20:48
  • $\begingroup$ @usεr11852 I only mention the GBDT model because it makes it clear that it is possible to do better than my current NN model using the given features. I'm not sure it matters, but the loss I'm optimizing is the LambdaRank loss. Basically it's a weighted pairwise training using binary cross entropy. $\endgroup$
    – dmh
    Commented Aug 24, 2020 at 20:51
  • $\begingroup$ I hadnt mentioned in my original post, but actually even the performance on the training set is worse for the NN model. Basically the NN model doesnt seem to be complex enough to fit the data, but increasing the depth or width of the network (as people usually suggest) does not help. $\endgroup$
    – dmh
    Commented Aug 24, 2020 at 20:54
  • 1
    $\begingroup$ That does not seem too perplexing. In general, I have found NN to be outperformed by GBMs in tabular data. Anyway, LambdaRank is available in LightGBM and other ranking function (e.g. LambdaMart, YetiRank, etc.) are available in XGBoost and Catboost too. $\endgroup$
    – usεr11852
    Commented Aug 24, 2020 at 21:14

1 Answer 1


A lot of the same techniques to training CNNs and other architectures also apply to training fully connected networks. A list of things that I tried and gave me much better performance:

  • Used SGD with a carefully selected learning rate and learning rate schedule. Using nesterov momentum with a momentum of 0.9 will also likely speed convergence. Adaptive methods are easy to configure, but SGD will do better when properly tuned [1].
  • Choose your learning rate based on a subset of the data with the batch size you want to use. SGD convergence is robust to the size of the dataset [5] section 1.
  • Use a smaller batch size, but also increase the initial learning rate. Some works have suggested a linear scaling rule, while others have suggested a square root scaling [2,3]. Warm up the learning rate from a smaller value [2]
  • If using LayerNorm, instead use a simple variant that does not learn bias and variance parameters (these tend to overfit and make the network more difficult to train)[4].
  • If using BatchNorm, be aware of how it is affected by distributed training [2]
  • Monitor network statistics every several iterations. Amount of activation saturation, gradient norms, ratio of the gradient norm to the weight norm and the weight norm can all be informative to how to further tune parameters.
  • Choose layer-wise initial learning rates. This can be done either using second order information (the inverse of the maximum eigenvalue of the hessian) or just first order information (ensure the ratio of the gradient norm to the weight norm is some common value such as 0.1) [5] chapter 1 and 18.
  • Local minimum in overparametrized networks are usually good enough [6].

Some personal anecdotal points:

  • Somehow, deeper networks are easier to optimize than wider networks
  • Bengio [5] section 19 claims that networks with fixed layer size work better than varying layer size. He also claims that using a wider layer size than the input dimension works better, although I have found that using a smaller size is easier to optimize.


[1] Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." Advances in neural information processing systems. 2017.

[2] Goyal, Priya, et al. "Accurate, large minibatch sgd: Training imagenet in 1 hour." arXiv preprint arXiv:1706.02677 (2017).

[3] Krizhevsky, Alex. "One weird trick for parallelizing convolutional neural networks." arXiv preprint arXiv:1404.5997 (2014).

[4] Xu, Jingjing, et al. "Understanding and Improving Layer Normalization." Advances in Neural Information Processing Systems. 2019.

[5] Orr, Genevieve B., and Klaus-Robert Müller, eds. Neural networks: tricks of the trade. Springer, 2003.

[6] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.