Normalizing input to DNN with structured data

Question

I have a structured dataset of around 100 gigs, and I am using DNN for classification. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.

Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have read many articles on normalization and all of them assume that data fit into the memory and conveniently calculate the mean and SD for each feature. But for most real-world datasets that is not the case.

So, How should one go about normalizing input features while loading data in batches and training model?

Answer 1

What I do in these occasions is that I compute the dataset mean and std beforehand. So I do a single pass over the whole dataset just to compute the mean and std. Then I use these two numbers to normalize each image before I feed them to the model.

Another option would be to use a running mean and std, which would converge to the true mean and std after the first epoch. This is more efficient as you don't have to do the computation beforehand, but you'll probably need to code this by hand.