Derivative of vector Y=XW with respect to matrix W

Question

In the first derivation of dL/dW, I use the rule for the derivative of a constant with respect to a matrix and then apply the chain rule.

\begin{gather*} Y\ =\ XW\ +\ B\\ X=\begin{bmatrix} x_{0} & x_{1} & x_{2} \end{bmatrix} ,\ Y=\begin{bmatrix} y_{0} & y_{1} \end{bmatrix} ,\ W=\begin{bmatrix} w_{00} & w_{01}\\ w_{10} & w_{11}\\ w_{20} & w_{21} \end{bmatrix} ,\ B=\begin{bmatrix} b_{0} & b_{1} \end{bmatrix} \end{gather*}

When it comes to the derivative with respect to a vector, the rules I found assume column vectors. Are the rules the same for row vectors? (For numerator layout, dY/dL is a column vector. However, they don't say that Y has to be a column vector, instead they say "If the numerator y is of size m and the denominator x of size n")

\begin{gather*} \left(\frac{\partial L}{\partial W}\right)^{T} =\begin{bmatrix} \frac{\partial L}{\partial w_{00}} & \frac{\partial L}{\partial w_{00}}\\ \frac{\partial L}{\partial w_{10}} & \frac{\partial L}{\partial w_{11}}\\ \frac{\partial L}{\partial w_{20}} & \frac{\partial L}{\partial w_{21}} \end{bmatrix} =\begin{bmatrix} \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{00}} & \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{01}}\\ \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{10}} & \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{11}}\\ \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{20}} & \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{21}} \end{bmatrix}\\ \\ Focus\ on\ one\ term:\\ y_{0} \ =\ w_{00} x_{0} +w_{10} x_{1} +w_{20} x_{2} \ +b_{0}\\ y_{1} \ =\ w_{01} x_{0} +w_{11} x_{1} +w_{21} x_{2} +b_{1}\\ \\ \frac{\partial Y}{\partial w_{00}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{00}}\\ \frac{\partial y_{1}}{\partial w_{00}} \end{bmatrix} =\begin{bmatrix} x_{0}\\ 0 \end{bmatrix}\\ \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial w_{00}} =\ \begin{bmatrix} \color{red}{\frac{\partial L}{\partial y_{0}}} & \color{red}{\frac{\partial L}{\partial y_{1}}} \end{bmatrix}\begin{bmatrix} x_{0}\\ 0 \end{bmatrix} =\color{red}{\frac{\partial L}{\partial y_{0}}} \ x_{0} \ +\ \color{red}{\frac{\partial L}{\partial y_{1}}} *\ 0\ =\color{red}{\frac{\partial L}{\partial y_{0}}} \ x_{0}\\ \\ \frac{\partial Y}{\partial w_{10}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{10}}\\ \frac{\partial y_{1}}{\partial w_{10}} \end{bmatrix} \ =\begin{bmatrix} x_{1}\\ 0 \end{bmatrix} ,\ \frac{\partial Y}{\partial w_{01}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{01}}\\ \frac{\partial y_{1}}{\partial w_{01}} \end{bmatrix} \ =\begin{bmatrix} 0\\ x_{0} \end{bmatrix} ,\ \frac{\partial Y}{\partial w_{11}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{11}}\\ \frac{\partial y_{1}}{\partial w_{11}} \end{bmatrix} \ =\begin{bmatrix} 0\\ x_{1} \end{bmatrix} ,\\ \frac{\partial Y}{\partial w_{20}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{20}}\\ \frac{\partial y_{1}}{\partial w_{20}} \end{bmatrix} \ =\begin{bmatrix} x_{2}\\ 0 \end{bmatrix} ,\ \frac{\partial Y}{\partial w_{21}} =\ \begin{bmatrix} \frac{\partial y_{0}}{\partial w_{21}}\\ \frac{\partial y_{1}}{\partial w_{21}} \end{bmatrix} \ =\begin{bmatrix} 0\\ x_{2} \end{bmatrix}\\ \\ Finally:\\ \left(\frac{\partial L}{\partial W}\right)^{T} =\begin{bmatrix} \frac{\partial L}{\partial y_{0}} \ x_{0} & \frac{\partial L}{\partial y_{1}} \ x_{0}\\ \frac{\partial L}{\partial y_{0}} \ x_{1} & \frac{\partial L}{\partial y_{1}} \ x_{1}\\ \frac{\partial L}{\partial y_{0}} \ x_{2} & \frac{\partial L}{\partial y_{1}} \ x_{2} \end{bmatrix} =\begin{bmatrix} x_{0}\\ x_{1}\\ x_{2} \end{bmatrix}\begin{bmatrix} \frac{\partial L}{\partial y_{0}} & \frac{\partial L}{\partial y_{1}} \end{bmatrix} =\ X^{T}\color{red}{\frac{\partial L}{\partial Y}} \end{gather*}

Is dY/dW, the derivative of a vector with respect to a matrix, a third degree tensor? Am I allowed to do the following derivation? (writing a 3d tensor as a vector of 2d matrices)

\begin{gather*} \frac{\partial L}{\partial W} =\ \color{red}{\frac{\partial L}{\partial Y}}\frac{\partial Y}{\partial W} =\begin{bmatrix} \color{red}{\frac{\partial L}{\partial y_{0}}} & \color{red}{\frac{\partial L}{\partial y_{1}}} \end{bmatrix}\begin{bmatrix} \frac{\partial y_{0}}{\partial W}\\ \frac{\partial y_{1}}{\partial W} \end{bmatrix} =\color{red}{\frac{\partial L}{\partial y_{0}}}\frac{\partial y_{0}}{\partial W} +\color{red}{\frac{\partial L}{\partial y_{1}}}\frac{\partial y_{1}}{\partial W}\\ =\ \color{red}{\frac{\partial L}{\partial y_{0}}}\begin{bmatrix} \frac{\partial y_{0}}{\partial w_{00}} & \frac{\partial y_{0}}{\partial w_{01}}\\ \frac{\partial y_{0}}{\partial w_{10}} & \frac{\partial y_{0}}{\partial w_{11}}\\ \frac{\partial y_{0}}{\partial w_{20}} & \frac{\partial y_{0}}{\partial w_{21}} \end{bmatrix}^{T} +\color{red}{\frac{\partial L}{\partial y_{1}}}\begin{bmatrix} \frac{\partial y_{1}}{\partial w_{00}} & \frac{\partial y_{1}}{\partial w_{01}}\\ \frac{\partial y_{1}}{\partial w_{10}} & \frac{\partial y_{1}}{\partial w_{11}}\\ \frac{\partial y_{1}}{\partial w_{20}} & \frac{\partial y_{1}}{\partial w_{21}} \end{bmatrix}^{T}\\ =\ \color{red}{\frac{\partial L}{\partial y_{0}}}\begin{bmatrix} x_{0} & 0\\ x_{1} & 0\\ x_{2} & 0 \end{bmatrix}^{T} +\color{red}{\frac{\partial L}{\partial y_{1}}}\begin{bmatrix} 0 & x_{0}\\ 0 & x_{1}\\ 0 & x_{2} \end{bmatrix}^{T} =\begin{bmatrix} \color{red}{\frac{\partial L}{\partial y_{0}}} x_{0} & \color{red}{\frac{\partial L}{\partial y_{1}}} x_{0}\\ \color{red}{\frac{\partial L}{\partial y_{0}}} x_{1} & \color{red}{\frac{\partial L}{\partial y_{1}}} x_{1}\\ \color{red}{\frac{\partial L}{\partial y_{0}}} x_{2} & \color{red}{\frac{\partial L}{\partial y_{1}}} x_{2} \end{bmatrix}^{T}\\ =\ \left( X^{T}\color{red}{\frac{\partial L}{\partial Y}}\right)^{T} \end{gather*}

Edit: Found a similar question, but the final answer is different.

Answer 1

The choice of layout notation and the treatment of vector variables (i.e. row/column) is usually tricky and may yield inconsistent results. So, it's hard to tell that the rules are the same for both. For example, in some cases, the multiplication terms in the chain rule goes from right to left, as opposed to what we're accustomed to, i.e. left to right.

For your second question, the matrix multiplication is not valid because the LHS, $\partial L/\partial Y$ has dimension $1\times 2$ and RHS, has dimension $4\times 3$. But, the expression

$$\frac{\partial L}{\partial W}=\frac{\partial L}{\partial y_0}\frac{\partial y_0}{\partial W}+\frac{\partial L}{\partial y_1}\frac{\partial y_1}{\partial W}$$

is correct due to chain rule in multivariate calculus. Therefore, the rest follows.

Answer 2

First note the differential relationship between $Y$ and $W$ $$\def\qiq{\quad\implies\quad} Y = XW+B \qiq dY = X\;dW $$

Let $L_Y$ denote the gradient of $L$ with respect to $Y$.

Write the differential of $L$ in terms of $Y,\,$ change variables from $Y\to W$ and isolate $L_W$ $$\eqalign{ dL &\equiv L_Y:dY \\ &= L_Y:(X\;dW) \\ &= (X^TL_Y):dW \\ dL &\equiv L_W:dW \qiq L_W &= X^TL_Y \\\\ }$$

---

$^{**}$A colon denotes the matrix inner product, i.e. $$\eqalign{ A:B &\;=\; \operatorname{trace}(A^TB) \\ }$$ The properties of the trace lead to these useful rules $$\eqalign{ A:B &\;=\; B:A \;=\; B^T:A^T \\ CA:B &\;=\; C:BA^T \\ }$$