Parameter optimization Neural network with small dataset

Question

I have a dataset with 30000 data points. I will use a neural network with resilient backpropagation to train it. I have two parameters to set for the resilient backprogation algorithm.

I wanted to use a subset of the dataset (use about 1500 data points) to tune the parameters. A validation dataset is included in the 1500 data points. The reason for this is that, tuning paramters with 30000 data points takes about 1 hour and to avoid local mimimum, I have to train it 10 time using different initialization weights.

Therefore it will take too much time. Can I use the small part of the dataset to tune the parameters as described above? Is it a good practice?

Answer 1

Typically it takes a lot of time to try different parameters for neural network. There are some advanced algorithms like Bayesian optimization or Tree-structured Parzen estimators (TPE). They can minimize number of trials that you need to do before selecting the best set of parameters. Basically they are able to learn a good parameters selection using historical data from previous trials. You can check this article to find more about these algorithms.

It's not a good thing to use less data to train a neural network. The reason is that networks are learnable feature extractors and they require a lot of training data to be able to learn good features. In addition, with small datasets you are more likely to overfit your network and as a result most of your trials can give high errors for validation dataset. You can decrease number of samples as long as you don't get significant decrease in accuracy (you can check it with your current model).

Some useful tips that can speed up your training:

• Use 32-bit float numbers instead of 64-bit
• Select a reasonable distribution for your hyperparameters. For instance, selecting number of hidden units from uniformal distribution between 100 and 10,000 is probably a bad choice, because values for 10,000 will more likely to give you improvement (with proper regularization) but it will take much longer to train your network.
• Optimize your code. For python you can check the line_profiler library. It can help you to find lines of code where program spends most of the computational time.