At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$.
However, now I need to derive ridge regression for the case that the target- and predicted values have the dimensions $1\times k$ with $k > 1$.
I have found these very useful links:
How to derive the ridge regression solution?
https://tamino.wordpress.com/2011/02/12/ridge-regression/
https://towardsdatascience.com/ridge-regression-for-better-usage-2f19b3a202db
https://en.wikipedia.org/wiki/Tikhonov_regularization
It seems to me that all of the above mentioned links also assume that the target- and predicted values have the dimensions $1\times1$.
Therefore, I am asking for help for the derivation of ridge regression for multi-value-target vectors.
I started the derivation by building the model equation.
$Y(W,X) = \Phi W = \begin{pmatrix} \phi_1(x_1) & \phi_2(x_1) & ... & \phi_m(x_1)\\ \phi_1(x_2) & \phi_2(x_2) & ... & \phi_m(x_2)\\ \vdots & \vdots & \ddots & \vdots\\ \phi_1(x_n) & \phi_2(x_n) & ... & \phi_m(x_n)\\ \end{pmatrix} \begin{pmatrix} w_{11} & w_{12} & ... & w_{1k}\\ w_{21} & w_{22} & ... & w_{2k}\\ \vdots & \vdots & \ddots & \vdots\\ w_{m1} & w_{m2} & ... & w_{mk}\\ \end{pmatrix} = \\ \begin{pmatrix} \sum_{j=1}^m w_{j1} \phi_j(x_1) & \sum_{j=1}^m w_{j2} \phi_j(x_1) & ... & \sum_{j=1}^m w_{jk} \phi_j(x_1) \\ \sum_{j=1}^m w_{j1} \phi_j(x_2) & \sum_{j=1}^m w_{j2} \phi_j(x_2) & ... & \sum_{j=1}^m w_{jk} \phi_j(x_2) \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{j=1}^m w_{j1} \phi_j(x_n) & \sum_{j=1}^m w_{j2} \phi_j(x_n) & ... & \sum_{j=1}^m w_{jk} \phi_j(x_n) \end{pmatrix} \\$
Please note that each row of $Y$ is one multi-value-prediction vector of the dimension $1\times k$. So, $Y$ has the dimension $n\times k$, where $n$ is the number of observations. $\phi_j$ is the $j$th of $m$ functions which takes $x_i \in R^{e}$ with $e \in N$ and calculates a single value.
Now, I edit equation 11 of my university slides to: $E_D(W) = \frac{1}{2}\bigg((\Phi W - Z) \odot (\Phi W - Z) + \lambda W \odot W \bigg)$ where $\odot$ is the Hadamard product and $Z$ is the matrix consisting of multi-value-target vectors.
I know that $W \odot W$ is not suitable here, but I have no idea what else is consistent to $||W||^2_2$ (from the slides).
Next, I want to adjust equation 7: $\nabla_W E_D(W) = \frac{\partial}{\partial W} \frac{1}{2}\bigg((\Phi W - Z) \odot (\Phi W - Z) + \lambda W \odot W \bigg) = \begin{pmatrix} 0 \\ \vdots \\ 0 \\ \end{pmatrix}$
And here is the point, where the problems start. How is this equation possible to solve?
I have no idea...
For several problems, I have validated that the following equation still holds, even with the assumption that $Z$ consists of multi-value-target vectors:
$W_{optimal} = \underbrace{(\Phi^T \Phi+\lambda I)^{-1}\Phi^T}_{\substack{\Phi^\dagger}}Z$
where $\Phi^\dagger$ is the so-called Moore-Penrose pseudo-inverse of $\Phi$.
Unfortunately, I cannot derive the last equation with the help of the before mentioned derivative of $E_D(W)$.
Can someone please help me?