7
$\begingroup$

I am currently trying to learn about Reproducing Kernel Hilbert spaces (RKHS) and would like to gain some intuition about its reproducing property. The RKHS is defined with kernel $k(x,x')$ which maps from a space $\mathbb{R}^N\times\mathbb{R}^N\rightarrow\mathbb{R}$, for $x,x'\in\mathbb{R}^N$. If we fix one of the kernel's arguments at a certain $x$ we shorten this as $k(x,\cdot)=k_x(\cdot)$. Now Wikipedia lists the reproducing property as

$$f(x)=<f,k_x>$$

I would like to understand what this means intuitively. In a paper I am currently studying, the authors use the reproducing property to obtain the following formula:

$$M(x)=\int{M(x')k(x,x')}dx'$$

where they assume the operation $M(x)\in\mathbb{R}^Y$ lies in a RKHS. I'm not quite certain how they obtain this result. Here is my attempt at explanation: A look at another StackExchange answer mentions the 'integral form of the inner product', which the stackexchange answer denotes as

$$<f,g>=\int{f(x)g(x)dx}$$

which looks a lot like

$$<f,k_{x'}>=\int{f(x')k(x',\cdot)dx'}$$

Is all of this correct so far? So if I now move away from the limit of infinity in Equation 2 and would take a Monte Carlo integral with a set of samples $x_i, i=1,...,N$, this would mean:

$$\hat{M}(x)\approx\frac{1}{S}\sum_{i=1}^{N}{M(x_i)k(x,x_i)}'$$

Or, more intuitively: does this mean that if I know $M(x_i)$ at locations $x_i$, then I could derive $M(x)$ simply by evaluating the kernel $k(x,x_i)$ (at least in the limit of $N\rightarrow\infty$, where $\hat{M}(x)=M(x)$)?


Continuing question: In the paper, the authors then use a set of $S\in\mathbb{N}$ Monte Carlo samples to 'generate a finite Hilbert space' and approximate the gradient of $M(x)$ in $\mathbb{R}^N$:

$$\nabla_xM(x)\approx\frac{1}{N}\sum_{i=1}^{S}M(x_i)\nabla_xk(x,x_i)$$

How do they arrive at this result? What I believe might have been done: they replaced $M(x)$ with $\nabla_xM(x)$ and then moved the Nabla operator to the kernel. Independently, they approximated the integral with Monte Carlo samples. Is this what happened here? If so, why may one freely move the Nabla operator around? If not, what is going on here instead?

$\endgroup$

1 Answer 1

4
$\begingroup$

You're overthinking this: the result with $M$ is just a restatement of the reproducing property. Using wikipedia's notation, we have that the reproducing property is defined as having an element $k_x \in \mathcal H$ for each $x\in\mathcal X$, the domain, such that $$ \langle f, k_x\rangle = f(x) $$ for each $f\in\mathcal H$.

Writing out the inner product as an integral w.r.t. a measure $\mu$ (I'm using Lebesgue integrals so this applies to both sums and the usual integrals) we have $$ f(x) = \langle f, k_x\rangle = \int_{\mathcal X} f(x')k_x(x')\,\text d\mu(x') \\ = \int_{\mathcal X} f(x')k(x,x')\,\text d\mu(x') $$ so plugging in $M=f$ exactly gives what they state.

$\endgroup$
5
  • $\begingroup$ Thank you very much for the answer - it seems that I have to read up some more on the relation between integrals and the inner product. $\endgroup$
    – J.Galt
    Sep 3, 2019 at 15:00
  • $\begingroup$ I have also expanded the question with a second part, showing their next result and maybe illustrating where and why I got confused - could you also shed some light on this, please, if it makes sense to you? $\endgroup$
    – J.Galt
    Sep 3, 2019 at 15:02
  • $\begingroup$ @J.Galt (1/2) without the paper i have to guess, but my guess is that they are picking representers $\{k_{x_1}, \dots, k_{x_S}\}$ and then $\hat M$, the approximation of $M$, is in the span of these $S$ functions. Assuming the kernel matrix is positive definite and therefore invertible, this means these representers give a basis for an $S$-dim space and we'll have $\hat M = \sum_{i=1}^S \alpha_i k_{x_i}$. If we know $\hat M(x_i)$, or maybe $M(x_i)$, then we can get $\hat\alpha$ by $$\hat M(x_j) = \langle \hat M, k_{x_j}\rangle = \sum_{i=1}^S \alpha_i k(x_i,x_j)$$ so $\endgroup$
    – jld
    Sep 4, 2019 at 18:34
  • $\begingroup$ (2/2) $\hat \alpha = K^{-1}\mathbf m$ where $\mathbf m$ are the observed values and $K$ is the $S\times S$ kernel matrix. Derivatives then distribute through by linearity for $\nabla \hat M$. This isn't exactly what they have but my guess is that they are doing something like this. $\endgroup$
    – jld
    Sep 4, 2019 at 18:35
  • $\begingroup$ Phew, wow! This definitely requires some more work on my end to understand, but your response give a great basis to start from - thank you a lot! $\endgroup$
    – J.Galt
    Sep 5, 2019 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.