2
$\begingroup$

I am studying matrix calculus for linear regression and machine learning and I would like to know exactly if the following calculations are correct:

Let $y=\sin(x+yz)$ and $r=\begin{bmatrix}x\\y\\z\end{bmatrix}$

Then the following is the gradient of $y$ i.e., the derivative of $y$ with respect the vector $r$ in denominator layout: $$\frac{\partial y}{\partial r}=\frac{\partial\sin(x+yz)}{\partial r}=\begin{bmatrix}\frac{\partial \sin(x+yz)}{\partial x}\\\frac{\partial\sin(x+yz)}{\partial y}\\\frac{\partial \sin(x+yz}{\partial z}\end{bmatrix}=\begin{bmatrix} \cos(x+yz)\\z\cos(x+yz)\\y\cos(x+yz)\end{bmatrix}$$

In numerator layout it would be:

$$[\cos(x+yz), z\cos(x+yz), y\cos(x+yz)]$$


Now, let $$\mathbf{y}=\begin{bmatrix} e^{xyz}\\x^2z\\yx\end{bmatrix}$$

So in numerator layout: $$\frac{\partial\mathbf{y}}{\partial r}=\begin{bmatrix} \frac{\partial e^{xyz}}{\partial x} & \frac{\partial e^{xyz}}{\partial y} & \frac{\partial e^{xyz}}{\partial z}\\ \frac{\partial x^2z}{\partial x} & \frac{\partial x^2z}{\partial y} & \frac{\partial x^2z}{\partial z}\\ \frac{\partial yx}{\partial x} & \frac{\partial yx}{\partial y} & \frac{\partial yx}{\partial z}\end{bmatrix}\\ =\begin{bmatrix} yze^{xyz} & xze^{xyz} & xye^{xyz}\\2xz& 0 & x^2 \\y & x & 0\end{bmatrix} $$

In denominator layout it would be the transpose of the above matrix?


Now, let

$$Y=\begin{bmatrix} x^2yz & xy^2z \\xyz^2 & \ln(xyz) \end{bmatrix}=\begin{bmatrix} Y_{11} & Y_{12} \\Y_{21} & Y_{22} \end{bmatrix}$$

Then $$\frac{\partial Y}{\partial r}=\begin{bmatrix} \frac{\partial Y_{11}}{\partial x} & \frac{\partial Y_{12}}{\partial x} & \frac{\partial Y_{11}}{\partial y} & \frac{\partial Y_{12}}{\partial y} & \frac{\partial Y_{11}}{\partial z} & \frac{\partial Y_{12}}{\partial z} \\ \frac{\partial Y_{21}}{\partial x} & \frac{\partial Y_{22}}{\partial x} & \frac{\partial Y_{21}}{\partial y} & \frac{\partial Y_{22}}{\partial y} & \frac{\partial Y_{21}}{\partial z} & \frac{\partial Y_{22}}{\partial z}\end{bmatrix}$$

Would this be correct? I'm worrying I mixed up the shape and/or the positions of the members of the matrices $\frac{\partial\mathbf{y}}{\partial r}$ and $\frac{\partial Y}{\partial r}$.

$\endgroup$
1
$\begingroup$

There are different layout notations and each one is correct if used consistently. So, the first one is correct under denominator layout because the output dimension follows the denominator variable's dimension. The second one is correct in numerator layout. But, the third one is wrong because the derivative of a vector wrt a matrix and vice versa is a 3D tensor, not a matrix.

$\endgroup$
2
  • $\begingroup$ In this video (towards 6:30): youtube.com/watch?v=WrH-jpJIqFQ , he says that it's a kronecker product between $\left(\frac{\partial}{\partial x}, \frac{\partial}{\partial y}, \frac{ \partial }{\partial z}\right)$ and $Y$ $\endgroup$
    – Slim Shady
    Aug 29 at 12:15
  • $\begingroup$ It can be unrolled as kronecker product, bur essentially it’s a 3D tensor. This is like flattening a matrix. $\endgroup$
    – gunes
    Aug 29 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.