1
$\begingroup$

I have a structured dataset of around 100 gigs, and I am using DNN for classification. Because of this huge dataset, I cannot load entire data in memory for training. So, I'll be reading data in batches to train the model.

Now, the input to the network should be normalized and for that, I need training dataset mean and SD. I have read many articles on normalization and all of them assume that data fit into the memory and conveniently calculate the mean and SD for each feature. But for most real-world datasets that is not the case.

So, How should one go about normalizing input features while loading data in batches and training model?

$\endgroup$
2
$\begingroup$

What I do in these occasions is that I compute the dataset mean and std beforehand. So I do a single pass over the whole dataset just to compute the mean and std. Then I use these two numbers to normalize each image before I feed them to the model.

Another option would be to use a running mean and std, which would converge to the true mean and std after the first epoch. This is more efficient as you don't have to do the computation beforehand, but you'll probably need to code this by hand.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.