2
$\begingroup$

I'm constructing an optimization (Bayesian optimization) algorithm using Java code. I have created the program, but the similarity values between inputted vectors in the kernel equation does not translate into the output similarity expected between vectors that should be "similar". I have a suspicion this has to do with the weighting of the differences between each of the components of the vectors because the parameter ranges of the different components are of completely different magnitudes (for example one parameter has a range 0.0 - 0.9 and another has a range of 100 - 500000).

I guess my question falls into two parts. First, how do I weight each the of the components of the input vectors evenly? Second, do I make the hyperparameters (width variable and sigma) vectors or scalar values?

I've been using this function I found from this other question (Which is helpful, but does not fully answer any of my questions): Kernels in Gaussian Processes

$$f(x_i,x_k)=σ^2 \exp\left(−\frac{1}{2 \ell^2} \sum_{j=1}^q (x_{i,j} − x_{k,j})^2 \right)$$

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

As you've written it here, $\sigma$ and $\ell$ are scalars. You could use a similar kernel, sometimes called an "Automatic Relevance Determination" (ARD) kernel, where $\ell$ is a vector of the same dimensionality as the data points:

$$f(x_i, x_k) = \sigma^2 \exp\left( - \frac{1}{2} \sum_{j=1}^q \left( \frac{x_{i,j} - x_{k,j}}{\ell_j} \right)^2 \right)$$

This allows hyperparameter optimization to select the right weight for each dimension. You're right to be concerned about using a single $\ell$ when one dimension has a range 500,000 times the other one: that kernel will "care" about the bigger dimension 500,000 times as much as it does about the smaller dimension.

A reasonable thing to do with your data is to standardize it so each dimension is on the same scale. This is the same thing as using an ARD kernel with each $\ell_j$ set to the product of some global scale $\ell$ and the standard deviation of the data in the $j$th dimension.

Improve this answer
$\endgroup$
13
  • $\begingroup$ I was playing around with some of the things that you said, and I decided to scale each of the dimensions by dividing the component differences by their parameter ranges. Basically, each difference is a ratio of the parameter dimension. I then took the sum of the squares of the ratios and replaced the sigma expression with the expression: p/(d-s) - (p/d) where p is a new parameter I call similarity because it governs how similar two vectors are, d is the dimension number of the vector, and s is the sum of the squares. It seems to be working, but is this mathematically sound? $\endgroup$
    – Cooper Scher
    Commented Aug 22, 2018 at 15:45
  • $\begingroup$ @Cooper Making sure I understand: you're using the kernel $$k(x, y) = \frac{p}{d - \sum_{i=1}^d \left( \frac{x_i - y_i}{\mathrm{max}_i - \mathrm{min}_i} \right)^2} - \frac{p}{d}?$$ If that's right, that's a problem; $k(x,x) = 0$ which means your "similarity" measure is actually a dissimilarity, and I don't think a GP is going to be sensible. Or are you doing $\sigma^2 \exp(- \mathrm{that})$? $\endgroup$
    – Danica
    Commented Aug 22, 2018 at 19:39
  • $\begingroup$ I'm doing the latter although I did keep the 1/2ℓ^2 term so its σ2exp(− 1/2ℓ^2 * that). Does that make sense mathematically? $\endgroup$
    – Cooper Scher
    Commented Aug 24, 2018 at 14:34
  • $\begingroup$ Ah, okay, this is far more likely to be legitimate. (Though note that you don't need both $p$ and $\ell$, they do the same thing....) It turns out that this is valid if and only if $\mathrm{that}$ is what's called a Hilbertian metric, i.e. there is some Hilbert space whose metric agrees with $\mathrm{that}$. I don't see an obvious way to check whether or not this is true in your case. One thing you can do: (...) $\endgroup$
    – Danica
    Commented Aug 24, 2018 at 14:43
  • $\begingroup$ (...) make some kernel matrices $K_{ij} = k(x_i, x_j)$ and check whether they're positive semi-definite, i.e. have no negative eigenvalues. You might get some -1e-8 eigenvalues from numerical error, but if there are any truly negative eigenvalues, then the kernel is not valid and your GP will not be well-posed. $\endgroup$
    – Danica
    Commented Aug 24, 2018 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.