3
$\begingroup$

I'm working on implementing a logistic regression algorithm in code. It's based this link. Unfortunately, the paper doesn't talk about weighting the individual examples $x_{i}$.

I think the relevant log likelihood function will look something like this:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} r_i $$

as opposed to what's in the paper:

$$ L(\vec{w}) = \sum_{i=1}^n \log{g(y_i z_i)} $$

where $z_i=\sum_k w_k x_{ik}$ and $r_i$ is the instance weight for the given instance $i$. Also, $y_i \in\{-1,1\}$ in this case, and $g$ is the sigmoid function so $1-g(z)=g(-z)$. This is discussed in the link.

Unfortunately, my math skills are not solid enough to be able to solve for the first and second partial derivatives, which are required to perform the optimization. Without the instance weights I'd like to add, the derivatives are:

$$ \frac{\partial{L}}{\partial{w_{k}}} = \sum_{i=1}^{n}y_{i}x_{ik}g(-y_{i}z_{i}) $$ $$ \frac{\partial^{2}{L}}{\partial{w_{j}}\partial{w_{k}}} = -\sum_{i=1}^{n}x_{ij}x_{ik}g(y_{i}z_{i})g(-y_{i}z_{i}) $$

How do these translate with the new $r_{i}$ instance weight involved?

Thanks!

$\endgroup$
  • 3
    $\begingroup$ All you need to understand is that multiplying anything by a constant value is tantamount to changing its units of measurement. That immediately shows that the rates of change (the derivatives) get multiplied by the same constant value. This principle applies term-by-term in the sum. $\endgroup$ – whuber Nov 30 '12 at 16:43
  • 1
    $\begingroup$ Do you mean $L(\vec{w}) = \sum_{i=1}^n r_i \log{g(y_i z_i)}$ or $L(\vec{w}) = \sum_{i=1}^n \log(r_i g(y_i z_i))$? In the first case, the $r_i$ go through to the gradient. In the second they go away in the gradient entirely. $\endgroup$ – Josephine Moeller Nov 30 '12 at 21:17
4
$\begingroup$

The weights $r_i$ are not a function of $w_i$. So when computing derivatives, you should treat them as a constant.

In particular, the partials w.r.t $w_i$ looks like:

$$ \frac{\partial{L}}{\partial{w_{k}}} = \sum_{i=1}^{n}r_i y_{i}x_{ik}g(-y_{i}z_{i}) $$ $$ \frac{\partial^{2}{L}}{\partial{w_{j}}\partial{w_{k}}} = -\sum_{i=1}^{n}r_i x_{ij}x_{ik}g(y_{i}z_{i})g(-y_{i}z_{i}) $$

If it helps you conceptually, you can think about problems with weighted samples as equivalent to unweighted problems, but where some particular observation $(x,y)$ appears $r$ times rather than only once.

$\endgroup$
  • 1
    $\begingroup$ Yeah, this is what I went with. To give a bit more detail, I'm trying to use a weighted logistic regressor as the 'weak learner' in an AdaBoost classifier. So I need some way of adjusting the classifier based on the AdaBoost weights. However, I'm not yet completely sure that just putting a linear weight on the log likelihood as I've proposed is the correct way to minimize the total misclassification error, which is what AdaBoost wants. $\endgroup$ – aardvarkk Dec 1 '12 at 3:53
  • $\begingroup$ I like that analogy "If it helps you conceptually, you can think about problems with weighted samples as equivalent to unweighted problems, but where some particular observation (x,y) ( x , y ) appears r r times rather than only once." $\endgroup$ – eggie5 Aug 20 '18 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.