R: neuralnet package is very slow and the final results are not good

Question

I am using neuralnet package to run artificial neural network on my data set for prediction. The response variable have 4 levels and I have 73 independent variables that are all dummy (indicator) variables and they are relatively sparse (I mean their values are mostly 0 rather than 1). My sample size is around 50000. When I run neuralnet with 1 hidden layer and 1 node, it takes like 10 minutes. I use a validation data set for validation and the prediction accuracy of this model is very low. Also I use more than 1 layer and more than 1 node and it takes hours to run and finally the prediction accuracy is still very bad. I used other methods for prediction such as multinomial logistic regression, K-nearest neighbour, SVM, random forests, etc. and they all have way better prediction accuracy. I use the default algorithm, I increase the step max to 1e+09 and increase the threshold to 0.1 and still takes hours and hours.

Can you tell me what is the appropriate number for hidden layers and affiliated number of nodes to get good prediction accuracy and also how can I make it converge faster. Thank you so much. Fred

Answer 1

This is more of a question for Cross-Validated, but I will answer here. Neural nets are not good models for sparse data. There are many reasons behind it but one of the reasons is that the optimization function for neural nets is not a nice parabola like for something such as a linear regression model, but rather a terrain with many peaks and valleys. In spares data, that terrain is basically flat, so as the model steps through your data trying to find optimal parameters, it is essentially wandering randomly hoping to get lucky and fall in a minima. This is also why it takes so long for your model to train, if there was a gradient it could follow, it would be able to converge more quickly, but since it is basically wandering, it takes a long time.

This is also part of the reason for the low accuracy. When your data isn't sparse, a good optimization method can "wiggle" out of a local minima, and descend to a better one, but here, if it get stuck in a local minima, there is basically no where else to go. When you add more nodes, you are expanding the search space for it to wander, making it take even longer to find something.

The problem here is not your number of hidden layers or number of nodes, it is your data. Either 1) you should give up on neural nets for this problem and use the other models where you got good results 2) you need to figure out how to transform your data to make it something more neural net compatible or 3) you need to work with a type of neural net designed to work with sparse data. Unless you have a really good reason why neural nets seem right for you (and not just that they are sexy right now), I would go with the first option. If you want to try transforming your data, PCA might be a place to start, though I am not an expert on this topic. If you want to try other networks, you will probably need to work in another language, because while R is lovely, well designed neural nets generally require specialized languages built specifically for that purpose like CNTK.