How can we use the technique of Lagrange multipliers to find a new vector of parameters $w$ which solves the optimization problem:
minimize J(w) = $\frac{1}{2} || w -u ||^2$
such that:
$w^T (x − y) ≥ 1$
Here $u$ is the current vector of parameters and $x$ and $y$ are two training examples such that $u^T(x-y) < 1$.
I rewrote the constraints as
$1 -wx -wy \leq 0 $
$ ux -uy -1 < 0$
Then the Lagrangian is,
$L = \frac{1}{2} (w-u)^2 + a(1-wx-wy) + b(ux-uy-1)$
where, $a,b \geq 0 $ are Lagrange multipliers.
But, when I set gradient to zero:
$0 = \frac{dL}{dw} = \frac{dL}{du} = \frac{dL}{dx} = \frac{dL}{dy} = \frac{dL}{da} = \frac{dL}{db} $
I got $w^* = 0 $, but I have to solve for w and I assume zero is not the correct answer here.
What am I doing wrong?
Update
Following the idea of jbowman, I got this,
$z = x-y$
minimize
J(w) = $\frac{1}{2} || w -u ||^2$
with constraint $1 - wz \leq 0$
Primal Lagrangian
$L_P = \frac{1}{2} (w-u)^2 + a(1-wz) $
$0 = \frac{dL}{dw} = \frac{dL}{du} = \frac{dL}{dz} = \frac{dL}{da} $
Another attempt,
I did not include dL/du and take only
$0 = \frac{dL}{dw} = \frac{dL}{dz} = \frac{dL}{da} $
Then I got
Dual Lagrangian,
$L_D = infimum_a L_P(w,u,z,a) = \frac{1}{2} a^2 z^2 +a $
Again, I got w = u.
This is problem of Large Margin Perceptron, but I got answer w equal to u. Whatever I initialize the values to u, the final weight vector w will be u.
Certainly, something is wrong. I am wondering what I did wrong?