Algorithms for weighted maximum likelihood parameter estimation

Question

What are the computational or algorithmic considerations for weighted maximum likelihood parameter estimation?

That is, I want to get $$ \theta^* = \arg\max\limits_\theta \sum_i w_i \log(\mathcal{L}(\theta|x_i)) $$ assuming we have a weight $w_i$ for each data point, such that $\sum_i w_i=1$. How is that generally done and are there alternative approaches to finding $\theta^*$?

References are appreciated in addition to full answers.

Answer 1

There are a number of ways to handle importance weights. Note that "weights" as a general term can be ambiguous. R's glm method, for instance, takes a weight parameter that is interpreted differently. This paper has a good discussion of a few approaches to handling importance weights.

• By far the most common approach when using stochastic optimisation methods is to just multiply each stochastic step by the importance weight for the sampled data point. This can work poorly if you have a mixture of very large and small weights. If there is less than a factor of 20 between your various weights it should work fine, although it might be slow to converge.
• Another approach when using SGD optimisation is rejection sampling, with probability proportional to $w_i/w_{\max}$. This is almost never used in practice though.
• Pre-sampling your dataset before applying a standard optimization algorithm is more common. Sample with replacement a new dataset with $w_i/w_{\max}$ proportional sampling. Typically you would take $2n$ to $10n$ samples, where n is the size of your original dataset.
• The linked paper suggests another approach which I believe is implemented in the Vowpal Wabbit package.

The popular liblinear package supports importance weights as well. If you're using LBFGS you can specify the loss and derivative manually, including the importance weights as you have in your post.