0
$\begingroup$

Please notice that this question was asked in MO, but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to this question.

Background:

For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ and $\overline{y}_i \in \mathbb{R}$. Then, for example, I could do the ridge-regression as given here link.

My question:

Can we do the same Kernel ridge regression but with a data set in matrix-vector form, i.e., $S := \{ X_i, y_i \}_{i=1}^{N}$, where $X_i \in M_{n,p}(\mathbb{R})$ and $y_i \in M_{n,1}(\mathbb{R})$? If yes, can you prove it or give me a reference/paper?

Thank you so much in advance for your help.

$\endgroup$
  • $\begingroup$ Although you describe a data structure, you haven't given a regression model. What is particularly lacking is any description of or assumptions about the distribution of the implicit errors: what can you tell us about that? In particular, if you are performing least squares regression, what is the structure of their covariance matrix? $\endgroup$ – whuber Oct 10 '18 at 14:19
  • $\begingroup$ If I have had the vector and scalar data set, then I could employ the minimization problem posed/mentioned here mathoverflow.net/questions/292843/… . Now, my question is how to extend such optimization problem in case I have matrix-vector data-set. One option is to vectorize the input matrix and consider vector-vector data set. $\endgroup$ – user550103 Oct 10 '18 at 19:18
2
$\begingroup$

Yes, sure.

To handle matrix-valued inputs $X_i$, you just need some appropriate kernel on matrices; the right choice will depend on your problem, but a simple one would be e.g. $k(X_i, X_j) = \exp\left( - \frac{1}{2 \sigma^2} \lVert X_i - X_j \rVert_F^2 \right)$. (This effectively throws away the matrix structure of the $X_i$ – it's equivalent to "flattening" them into vectors and using a Gaussian kernel on vectors. Unfortunately just plugging in a different norm than the Frobenius is not necessarily valid; the norm must be Hilbertian, i.e. correspond to the metric of some Hilbert space, for that to always work.)

For vector-valued outputs, you can extend the algorithm to operate with an output kernel as well. The paper

Álvarez, Rosasco, and Lawrence (2011). Kernels for Vector-Valued Functions: a Review.

gives an overview, and the landmark theoretical reference is

Caponnetto and de Vito (2007). Optimal Rates for the Regularized Least-Squares Algorithm. Foundations of Computational Mathematics, July 2007, Volume 7, Issue 3, pp 331–368.

The simplest choice corresponds to running a separate regression for each coordinate of $y$, but choosing an output kernel allows you to learn correlations in the output space as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.