0
$\begingroup$

How can I derivate the following optimization function?

$$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$

I just want to get the equations of the gradient descent method, so I want to get the partial derivative of $L$ with respect to $v$ and the partial derivative of $L$ with respect to $x$.

I just know how to calculate the first of the term of the function, but I have trouble to derive the second addend of the function:

$$\delta v_i^k L=-2\sum_i(y_{u,i}-v_i x_u)x_u^k + ¿?$$ $$\delta x_i^k L=-2\sum_i(y_{u,i}-v_i x_u)v_u^k + ¿?$$

$\endgroup$
17
  • 1
    $\begingroup$ You mention partial derivatives regarding $u$, but there is no $u$ in the expression $\endgroup$
    – Firebug
    Oct 31, 2020 at 18:02
  • $\begingroup$ Thanks! I mean x. Sorry for the misstake. $\endgroup$ Oct 31, 2020 at 18:03
  • $\begingroup$ If you're uncertain about vector notation, you might be able to make headway by re-writing it in terms of scalar products and sums. $\endgroup$
    – Sycorax
    Oct 31, 2020 at 18:12
  • 1
    $\begingroup$ What's the nature of $v_i$ and $x_u$? I take it they are vectors, but their product would be incompatible. Is $y_{u,i}$ a scalar? $\endgroup$
    – Firebug
    Oct 31, 2020 at 18:15
  • 2
    $\begingroup$ But you still didn't respond. I think you are missing a transpose in either $x_u$ or $v_i$, depending on if $y_{u,i}$ is a scalar or a matrix. The matrix product $x_uv_i$ does not exist for equal sized vectors, whereas the products $x_u^Tv_i$ and $x_uv_i^T$ do. $\endgroup$
    – Firebug
    Nov 1, 2020 at 1:34

1 Answer 1

1
$\begingroup$

As I mentioned in the long comment chain under OP, assuming $y_{u,i}$ is scalar, $v_i$ is a row vector and $x_u$ is a column vector (with matching sizes), we can show that:

$$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$

$$\frac{\partial L}{\partial x_a} =\frac{\partial }{\partial x_a}\left\{\sum_{i}(y_{a,i}-v_ix_a)^2+\lambda\|x_a\|_2^2\right\}\\ =\frac{\partial }{\partial x_a}\left\{\sum_{i}(y_{a,i}-v_ix_a)^2\right\}+\lambda\frac{\partial }{\partial x_a}\left\{\|x_a\|_2^2\right\}\\ =-2\sum_{i}(y_{a,i}-v_ix_a)v_i^T+2\lambda x_a\\ $$

By similar analogy

$$\frac{\partial L}{\partial v_j} =-2\sum_{u}(y_{u,j}-v_jx_u)x_u^T+2\lambda v_j\\ $$

Notice that both gradients were assumed (by me) to have the same dimensions of the parameters. Bar any error, you were missing only the derivative of the squared $\ell_2$ norm of a vector. This can be simply be shown to be, for vector $w$ (irrespective of the nature of $w$, be it a column or row matrix).

$$\frac{\partial \|w\|_2^2}{\partial w}=2w$$

Addendum: if you take vector derivatives to have a specific configuration, e.g. always rows or always columns, then some adjustments are due. Since you did not assume vectors to be column matrices (as we can see by the vector product), I think my solution is on point.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.