4
$\begingroup$

I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in the training set. The dataset is structured, ie, not images. My original features were highly right skewed; I do my best to transform these into nicer distributions by either taking logs or categorising some of them. I use an architecture of 512->128->128->1, with relu activations in every layer except the last. My loss function is sigmoid cross entropy.

The validation set contains 10 million rows. Initially the validation error goes down, but then starts to go up again after a couple of epochs. On analysing the gradients and weights of each layer, I see that the overfitting coincides with the weights on the final layer only getting larger and larger. The final layer seems to go into overdrive while the rest of the network seems to do very little learning.

I can solve the overfitting problem by using l2 regularisation, but this hurts the validation error. I have yet to find a beta regularisation parameter which doesn't hurt the best validation error I've seen. Dropout makes things even worse.

Granted, the classification problem is very difficult, with probably a very weak signal, but I find that gradient boosted trees are able to generalize much better than a simple, say, 64x64 multi-layer perceptron (the log loss on the training set is the same for both network and gradient boosted tree).

Are there any words of wisdom on how to make this network generalize better given that I've already tried:

  • dropout of varying degrees
  • l1/l2/group lasso regularization
  • adding noise to inputs
  • adding noise to gradients and weights
  • feature-engineering so as to remove/re-represent highly skewed features
  • batch normalization
  • using a lower learning rate on the final layer
  • simply using a smaller network (this is the best solution I've found)

to some or all layers. All methods hurt the validation error so much that the performance is nowhere near how well the tree model does. I would have given up by now were it not for the fact that the tree model is able to do so much better out of sample, but the training log loss for both is the same.

$\endgroup$
  • $\begingroup$ How big is your data set? How many inputs does the network have? Which activation function are you using? $\endgroup$ – CodesInChaos Oct 23 '17 at 11:15
  • $\begingroup$ I have 30 million rows and 500 features in the training set. The validation set contains 10 million rows. $\endgroup$ – housecat64 Oct 23 '17 at 12:10
4
$\begingroup$

With sample size $N=30\times10^6$ and 500 features, you already tried (most of) the usual regularization tricks, thus it doesn't look like there's much left to do at this point.

However, maybe the problem here is upstream. You haven't told us what's your dataset, exactly (what are the observations? What are the features?) and what are you trying to classify. You also don't describe in detail your architecture (how many neurons do you have? which activation functions are you using? What rule do you use to convert the output layer result into a class choice?). I will proceed under the assumptions that:

  • you have 512 units in input layer, 512 units in each of the hidden layers and 2 units in the output layer. corresponding to $p=525312$ parameters. In this case, your data set seems large enough to learn all weights.
  • you're using One-Hot Encoding to perform classification.

Correct me if my assumptions are wrong. Now:

  1. if you have structured data (this means you're not doing image classification), maybe there's just nothing you can do. Usually XGboost just beats DNNs on structured data classification. Have a look the Kaggle competitions: you'll see that for structured data, usually the winning teams use ensembles of extreme gradient boosted trees, not Deep Neural Networks.
  2. if you have unstructured data, then something's weird: usually DNNs dominate XGboost here. If you're doing image classification, don't use an MLP. Mostly everyone now uses a CNN. Also, be sure you don't use sigmoid activation functions, but stuff such as ReLU.
  3. You didn't try early stopping and learning rate decay. Early stopping usually "plays nice" with most other regularization methods and it's easy to implement, so that's the first thing I'd try, if I were in you. In case you're not familiar with early stopping, read this nice answer: Early stopping vs cross validation
  4. If nothing else helps, you should check for errors in your code. Can you try to write unit tests? If you're using Tensorflow, Theano or MXNet, can you switch to an high level API such as Keras or PyTorch? One might expect that using an high level API, where less customization is possible, would drive your test error up, not down. However, often the opposite happens, because the higher level API allows you to do the same work with much less code, and thus much less opportunity for mistakes. At the very least, you can be sure your high test error isn't due to coding bugs....

Finally, I didn't add anything about dealing with class imbalance because you seem quite knowledgeable, so I assume you used the usual methods to deal with class imbalance. In case I'm wrong, let me know and I'll add a couple tricks, citing questions dealing specifically with class imbalance if needed.

$\endgroup$
  • $\begingroup$ Thanks for your comments. I have updated the original post with the details you asked for.The other thing I want to note is that the training error always gets to a minimum within about 10 epochs. Then it just bobbles around. Also, I always get the same proportion of positives correct, even though the recall can vary. I find this very strange. $\endgroup$ – housecat64 Oct 23 '17 at 13:02
  • 1
    $\begingroup$ Hmmm, it's indeed strange. I don't think I can help more, but maybe adding a plot of the training and test set errors could attract more answers. Since you're using just one output neuron, you're using a threshold to perform classification, right? Did you tune the threshold using cross-validation and AUC? With structured data, it might just be the case that XGboost is better...it wouldn't be the first time I see this. However, this doesn't explain why the training set error stalls: this seems to indicate a bug in the code. By the way, these are not time series data, right? $\endgroup$ – DeltaIV Oct 23 '17 at 13:13
  • 1
    $\begingroup$ Ah! Then an MLP is really not a good architecture for time series. No wonder you're getting all these problems! Look into LTSM or RNN architectures. PS don't try downsampling...errors are correlated and you haven't studied their ACF, so it's not like downsampling every $N$ samples with an arbitrary $N$ will work. Rather than trying to force a NN architecture born to deal with iid data, to work on time series data, make the investment and go for architecture whose goal is to deal with time-dependent data :) $\endgroup$ – DeltaIV Oct 23 '17 at 14:02
  • 1
    $\begingroup$ I didn't not share that intuition because some of the "intuitions" in Deep Learning are folk tales: for example, the commonly held assumption that in image classification successive layers learn more "human-like" feature has been proven false by adversarial examples. Anyway, since you insist....The folk explanation is that since each tree is built on the residuals of the preceding ones, boosted trees are able to do better in problems where the errors have structure, such as time series, where errors are correlated. I don't have references for that right now, but I can point to .... $\endgroup$ – DeltaIV Oct 24 '17 at 13:31
  • 1
    $\begingroup$ ....this post by Peter Ellis where it seems that extreme gradient boosting outperforms a simple MLP on time series forecasting. However, the forecastxgb R package is still far from a CRAN submission, and it's hard to tell if results can be considered conclusive at this point. $\endgroup$ – DeltaIV Oct 24 '17 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.