Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Question

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to this question.

Background:

For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ and $\overline{y}_i \in \mathbb{R}$. Then, for example, I could do the ridge-regression as given here link.

My question:

Can we do the same Kernel ridge regression but with a data set in matrix-vector form, i.e., $S := \{ X_i, y_i \}_{i=1}^{N}$, where $X_i \in M_{n,p}(\mathbb{R})$ and $y_i \in M_{n,1}(\mathbb{R})$? If yes, can you prove it or give me a reference/paper?

Thank you so much in advance for your help.

Answer 1

Yes, sure.

To handle matrix-valued inputs $X_i$, you just need some appropriate kernel on matrices; the right choice will depend on your problem, but a simple one would be e.g. $k(X_i, X_j) = \exp\left( - \frac{1}{2 \sigma^2} \lVert X_i - X_j \rVert_F^2 \right)$. (This effectively throws away the matrix structure of the $X_i$ – it's equivalent to "flattening" them into vectors and using a Gaussian kernel on vectors. Unfortunately just plugging in a different norm than the Frobenius is not necessarily valid; the norm must be Hilbertian, i.e. correspond to the metric of some Hilbert space, for that to always work.)

For vector-valued outputs, you can extend the algorithm to operate with an output kernel as well. The paper

Álvarez, Rosasco, and Lawrence (2011). Kernels for Vector-Valued Functions: a Review.

gives an overview, and the landmark theoretical reference is

Caponnetto and de Vito (2007). Optimal Rates for the Regularized Least-Squares Algorithm. Foundations of Computational Mathematics, July 2007, Volume 7, Issue 3, pp 331–368.

The simplest choice corresponds to running a separate regression for each coordinate of $y$, but choosing an output kernel allows you to learn correlations in the output space as well.