3
$\begingroup$

Im trying to estimate the hyperparameters $\sigma_f \in \mathbb{R}$ and $\textbf{L} \in \mathbb{R}^d$ for an anisotropic squared exponential kernel

$$ k(\textbf{x},\textbf{x}') = \sigma_f^2 \cdot \exp{\left ( -\dfrac{1}{2}\sum_{i = 1}^{d}\left ( \frac{x_i-x_i'}{L_i} \right )^2 \right )},\; \textbf{x} \in \mathbb{R}^d , \; \textbf{x}' \in \mathbb{R}^d $$ regarding Gaussian Process regression by (the standard approach) minimizing the log marginal likelihood. This minimization is extremely sensitive to the choice of the start values and it often happens that either the minimization fails or the model fit is bad.

Is there any chance to find reasonable values from the given data ?

$\endgroup$
2
  • $\begingroup$ Can you give more details on what's going wrong? What's $n$ and $d$ in your case? What optimizer are you using? Also, do the different features have very different variances? $\endgroup$
    – jld
    Apr 6, 2021 at 20:55
  • $\begingroup$ The biggest problems are the overfitting of the regression and or the divergence of the minimization. d is the dimension of my problem, i.e. the dimension of the input data. As minimizer I use L-BFGS-B from the scipy package where I minimize the negative log marginal likelihood. $\endgroup$
    – user67080
    Apr 7, 2021 at 6:40

1 Answer 1

4
$\begingroup$

While using gradient-based optimizers like L-BFGS-B, you need to ensure that you choose good enough initialization points for the optimizer. This helps in efficient minimization of the NLL and also prevents early stopping.

For instance, you can use a grid-search-based approach (recommended in this article) to initialize the lengthscale ($𝐿_𝑖$) values and use the corresponding Generalized Least Square (GLS) solutions for the initialization of the variance ($𝜎^2_𝑓$). Finally choose the one from the grid, that yields the best likelihood value.

(NB: This initialization strategy is the default setup in the MATLAB/GNU Octave toolbox STK.)

$\endgroup$
3
  • 1
    $\begingroup$ Thanks for the link to the paper, I've experienced many of the same numerical problems with hyper-parameter optimisation with GPs and kernel machines. I tend to use Nelder-Mead simplex rather than grid search+gradient descent. Provided the number of hyper-parameters isn't too large it is about as quick (more iterations, but doesn't need gradient calculation) and has fewer numerical issues. $\endgroup$ Apr 7, 2021 at 9:27
  • 1
    $\begingroup$ I should add, the numerical problems in hyper-parameter optimisation tend to be even worse for GP classifiers... scope for a second paper? ;o) $\endgroup$ Apr 7, 2021 at 9:31
  • $\begingroup$ Thank you very much for the paper ! The reparameterization of the covaraince function seems promising ! $\endgroup$
    – user67080
    Apr 7, 2021 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.