# Kernel ridge regression with matrix-vector data set $S := \{ X_i, y_i \}_{i=1}^{N}$?

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to this question.

Background:

For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $$\overline{S}:= \{x_i, \overline{y}_i \}$$, where $$x_i \in M_{n,1}(\mathbb{R})$$ and $$\overline{y}_i \in \mathbb{R}$$. Then, for example, I could do the ridge-regression as given here link.

My question:

Can we do the same Kernel ridge regression but with a data set in matrix-vector form, i.e., $$S := \{ X_i, y_i \}_{i=1}^{N}$$, where $$X_i \in M_{n,p}(\mathbb{R})$$ and $$y_i \in M_{n,1}(\mathbb{R})$$? If yes, can you prove it or give me a reference/paper?

• Although you describe a data structure, you haven't given a regression model. What is particularly lacking is any description of or assumptions about the distribution of the implicit errors: what can you tell us about that? In particular, if you are performing least squares regression, what is the structure of their covariance matrix?
– whuber
Oct 10, 2018 at 14:19
• If I have had the vector and scalar data set, then I could employ the minimization problem posed/mentioned here mathoverflow.net/questions/292843/… . Now, my question is how to extend such optimization problem in case I have matrix-vector data-set. One option is to vectorize the input matrix and consider vector-vector data set. Oct 10, 2018 at 19:18

Yes, sure.

To handle matrix-valued inputs $$X_i$$, you just need some appropriate kernel on matrices; the right choice will depend on your problem, but a simple one would be e.g. $$k(X_i, X_j) = \exp\left( - \frac{1}{2 \sigma^2} \lVert X_i - X_j \rVert_F^2 \right)$$. (This effectively throws away the matrix structure of the $$X_i$$ – it's equivalent to "flattening" them into vectors and using a Gaussian kernel on vectors. Unfortunately just plugging in a different norm than the Frobenius is not necessarily valid; the norm must be Hilbertian, i.e. correspond to the metric of some Hilbert space, for that to always work.)

For vector-valued outputs, you can extend the algorithm to operate with an output kernel as well. The paper

Álvarez, Rosasco, and Lawrence (2011). Kernels for Vector-Valued Functions: a Review.

gives an overview, and the landmark theoretical reference is

Caponnetto and de Vito (2007). Optimal Rates for the Regularized Least-Squares Algorithm. Foundations of Computational Mathematics, July 2007, Volume 7, Issue 3, pp 331–368.

The simplest choice corresponds to running a separate regression for each coordinate of $$y$$, but choosing an output kernel allows you to learn correlations in the output space as well.