0
$\begingroup$

I am running a Kernel Ridge Regression in R. Mathematically, the minimization problem to be solved is the following:

$$ \min_{\boldsymbol{\beta} \in \mathbb{R}^{d}} \ \sum_{i = 1}^{n} (y_{i} - \left \langle \boldsymbol{\beta}, \phi(\boldsymbol{x_{i}}) \right \rangle )^{2} + \lambda \left \| \boldsymbol{\beta} \right \|^{2}$$

In particular, I found the function krr() from the listdtr package (here official documentation) to be particularly interesting. Indeed, a part from its efficiency, considering the Gaussian Kernel, it estimates a specific $\gamma_{j}$ for each of the $k$ variables. Considering for instance two individuals, $\boldsymbol{x_{i}}$ and $\boldsymbol{x_{j}}$, the associated Gaussian Kernel is:

$$K(\boldsymbol{x_{i}}, \boldsymbol{x_{h}}) = exp \left \{ -\sum_{j = 1}^{k}\gamma_{j}(x_{i,j} - x_{h,j})^{2} \right \}$$

where, for sake of completeness of the description:

$$\left \langle \phi(\boldsymbol{x_{i}}),\phi(\boldsymbol{x_{h}}) \right \rangle = K(\boldsymbol{x_{i}}, \boldsymbol{x_{h}})$$.

In presenting the code, please note that "Holdout" is the set used to train the model and estimate lambda and gamma, 35% of the dataset, while "estim" the remaining 65% to compute the MSE.

The code to perform the computation is the following:

library(listdtr)
set.seed(42)
krr_values <- krr(x = as.matrix(X_34_holdout), y = as.matrix(y_34_holdout))
test_predict_krr <- predict(krr_values, as.matrix(X_34_estim))
MSE_KRR_each_gamma <- colMeans((y_34_estim - test_predict_krr)^(2))

Now, I have a few questions: consider that source code is not available (or at least I could not find it):

1) In determining lambda and the gammas, what are the hyperparameter sets considered?

2) I understand I can extract the gammas (one for each of my 8 features) via:

krr_values$model$gamma
#realnumber1  realnumber2 realnumber3  realnumber4  realnumber5  
#realnumber6 realnumber7  realnumber8

Problem now is, how do I know each one of them to which of the variables do relate? For instance, one of them is about 20, but running the same script with two different seeds, once I estimated an 18 in was in the fourth position and another time about 20 in the second position: it is clearly the $\gamma_{j}$ associated with the same variable, but how can I then associate all of the others?

It would be particularly helpful the reply from someone who has already some experience with the aforementioned function, but any comment or reply will be highly appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.