Stochastic gradient descent (SGD) on data with weights

Question

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the common case in most research papers of not all.

However, what if the data records are not equally weighted? I have a dataset in which some records are of importance 1000 and others just 1-10. Pulling them randomly in a fixed sized mini-batch will result in different total weights in one batch to another. This will make the gradient of some batch gigantically large and also change the overall importance of samples.

One way to handle this is to "flatten" all the data records, for instance, duplicating a record of weight 10 to be 10 separate records and shuffled together with others with weight 1. But this will make the overall input data size 10x or more in size.

Is there any other mathematically smart way to deal with this issue? Any reference will be appreciated!

Answer 1

You are close to a reasonable solution. Instead of duplicating your data so that the uniform sampling respects the weights, you can sample the data according to its weights directly.

Suppose you have $n$ data points, $x_i$, for $i = 1, 2, ..., n$, with weights $w(x_i)$. Then, when you sample a mini batch, the probability that any one data point gets sampled is

$p(x_i) = \frac{w(x_i)}{\sum_{i=1}^{n}w(x_i)}$

Note that these are just the normalized weights.

You will need an algorithm or library that allows you do a weighted sampling, given the probabilities of each of the items. If you are using Python, you can use numpy.random.choice, where the array a might be indices into your data, and p would be the associated probabilities.