What are the rules of thumb for neural network configurations with large number of samples? I have a dataset with 200k samples, 400 features, and binary label classification problem. How should I choose the hidden layer size? I'm concerned that during training the network will gradually forget the older samples, and will overfit over the samples fed to it recently. How can I make sure this doesn't happen, and the network nicely generalizes over entire data set? Particularly I'm interested in choosing the right number of hidden layers (probably 1 is enough), number of hidden nodes, minibatch size, etc.
Q: How should I choose the hidden layer size?
Q: I'm concerned that during training the network will gradually forget the older samples, and will overfit over the samples fed to it recently. How can I make sure this doesn't happen, and the network nicely generalizes over entire data set?
If that happens, make sure you randomly draw your batches, e.g. 1:
As for any stochastic gradient descent method (including the mini-batch case), it is important for efficiency of the estimator that each example or minibatch be sampled approximately independently. Because random access to memory (or even worse, to disk) is expensive, a good approximation, called incremental gradient (Bertsekas, 2010), is to visit the examples (or mini-batches) in a fixed order corresponding to their order in memory or disk (repeating the examples in the same order on a second epoch, if we are not in the pure online case where each example is visited only once). In this context, it is safer if the examples or mini-batches are first put in a random order (to make sure this is the case, it could be useful to first shuffle the examples). Faster convergence has been observed if the order in which the mini-batches are visited is changed for each epoch, which can be reasonably efficient if the training set holds in computer memory.
1 Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.
Q: I'm interested in choosing the right minibatch size
You can compare the sequential vs. random access speed of your disk with CrystalDiskMark:
Crucial 512 GB m4 2.5-Inch Solid State Drive SATA 6Gb/s CT512M4SSD2:
Crucial M500 960GB SATA 2.5-Inch 7mm (with 9.5mm adapter/spacer) Internal Solid State Drive CT960M500SSD1:
A good explanation of the tests: http://www.overclock.net/t/1231707/can-someone-explain-the-different-crystaldiskmark-tests#post_17508715
Sequential: Crystal disk mark (CDM) reads/writes whatever file size you choose when you start the test sequentially. That is to say it starts writing on a sector and then writes the next part on the adjacent sector and so on. This is fastest because the head doesn't have to move about a lot as all the sectors are adjacent.
512k: CDM read/writes to random sectors on the drive, but it reads/writes 512KB of data at a random point, then moves to the next random point. This is faster than 4k because there's more data read/written with less movement of the head.
4k: The same as above but instead of reading/writing the test data in 512KB 'chunks' it reads/writes in 4KB chunks.
4kQD32: The same as 4K but there are more requests for the data sent to the HDD controller. I'm told that some HDDs increase performance when this happens because of the way their controller logic works but I think this mostly applies to SSDs not mechanical drives.