0
$\begingroup$

In Gaussian Processes, we have kernel parameters. For example, for RBF kernel, we have $k(x, x')= \sigma_f^2 \exp(-\frac1{2l^2}(x-x')^2)$, where the $\sigma_f, l$ are the two parameters.

My questions are:

  1. Are they considered hyperparameters or parameters? I feel they are because they seem not attached to the model learning the data, but I cannot articulate it very well.
  2. If they are hyperparameters, then why do I see that their values are often estimated through maximizing the marginal likelihood. For example, in Kevin Murphy's Machine Learning book, or in the fit() function of scikit-learn.GaussianProcessRegressor here. Aren't hyperparameters not learnable and need to be found through grid search or random search?
$\endgroup$
2
  • $\begingroup$ Kernels are estimated from data all the time. Whether to call them "hyperparameters" is a matter of preference. Hyperparameters in the Bayesian sense are also estimated from data, oftentimes. $\endgroup$
    – stans
    Jan 26 at 3:11
  • $\begingroup$ Why are kernel parameters for SVM seem not estimated from data? Also, kernel bandwidth for kernel smoothing/density estimation seems not estimated from data either. $\endgroup$ Jan 26 at 4:12
2
$\begingroup$

Strictly speaking the parameters of a Gaussian process are the mean function and covariance kernel (they are what "parametrises" or "defines" this distribution). The parameters of the mean function and covariance kernel are then "parameters of these parameters" hence "hyperparameters" in most texts.

In the RBF kernel example you provide, $\rho_f$ and $l$ are parameters of the kernel, and hyperparameters of any Gaussian process which uses that particular kernel as a parameter.

This should hopefully answer your second question, too; for something to be a "hyperparameter" (at least in the context of GP regression) simply means it is a parameter of either the mean function or covariance kernel. It does not -- to my knowledge -- have anything to do with how "that thing" (that is, the hyperparameter) might be inferred or estimated for given data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.