2
$\begingroup$

I've read from various sources that Hyperbolic Tangent kernels are not positive semi-definite and thus are not actually a valid kernel. Does this mean they are misnomer?

Furthermore, if they are technically not a kernel, how come they perform well in certain cases? What type of data do they perform well?

And what is the rationale behind their method of measuring similarity? By this I mean: as we can consider the rationale behind polynomial kernels to be taking the logical AND of the features of a feature vector, what is the corresponding analogy for tanh kernels?

Hyperbolic Tangent kernels are sometimes also called Sigmoid Kernels or tanh kernels and are defined as $$ k(x,x^\prime)=\tanh\left(\nu+ x\cdot x^\prime\right) $$ This website provides some discussion.

$\endgroup$
3

2 Answers 2

6
$\begingroup$

I've read from various sources that Hyperbolic Tangent kernels are not positive semi-definite and thus are not actually a valid kernel. Does this mean they are misnomer?

Consider $X=\{x_1,x_2\}.$ Denote $K$ the Gram matrix of $X$ for $k$ the hyperbolic tangent kernel. It has eigenvalues given by the set of solutions $\lambda$ to

$$\begin{align} 0 &= |I\lambda-K| \\ &= \begin{vmatrix} \lambda-k_{11} & k_{12} \\ k_{21} & \lambda-k_{22} \\ \end{vmatrix} \\ &=\lambda^2-\lambda(k_{11}+k_{22})+k_{11}k_{22}-k_{12}k_{21} \end{align} $$

Clearly this is a quadratic. Observe that this definition of the hyperbolic tangent kernel is symmetric because the dot product is symmetric, so $k_{12}=k_{21}$, so we know $$\begin{align} \lambda &= \frac{(k_{11} + k_{22})\pm\sqrt{(-k_{11} - k_{22})^2-4\times1\times (k_{11}k_{22} - k_{12}k_{21})}}{2 \times 1} \\ &= \frac{1}{2}\left[ (k_{11} + k_{22}) \pm \sqrt{(k_{11} - k_{22})^2 - 4k_{12}k_{21}} \right] \end{align} $$

A matrix is PSD if and only if all $\lambda\ge0$, but specific choices of $\nu, x_1, x_2$ will result in negative $\lambda$. Showing indeterminacy is a matter of picking $k_{ij}$ s.t. at one or two solutions $\lambda$ are negative, and then backing into values $x$ which yield those $k_{ij}$.

if they are technically not a kernel, how come they perform well in certain cases? What type of data do they perform well?

It may be that they are only "slightly" non-PSD in cases where they perform well, by which I mean that the negative eigenvalues are "close to" 0. Or it may be that the most important attributes of RKHS are retained even in the case of indefinite Gram matrices, and losing PSD is only a slight reduction in power. Finding cases where $k$ works well is just a matter of finding a paper that puts it to use. The website you linked to provides one such example. But I'm not convinced that there are general classes of kernels that are better for some data as compared to other data that is distinct from choices of how to represent feature vectors. Indefinite kernels are an ongoing topic of research.

And what is the rationale behind their method of measuring similarity? By this I mean: as we can consider the rationale behind polynomial kernels to be taking the logical AND of the features of a feature vector, what is the corresponding analogy for $\tanh$ kernels?

In this definition of the $\tanh$ kernel, we only look to the inner product of the feature vectors. These might be considered as "non-nomralized" cosine similarities, so they are larger when two vectors are more similar, and negative if they are dissimilar, so we have $\tanh(10)\approx 1$ and $\tanh(-10)\approx -1$. The parameter $\nu$ shifts where the sign changes away from 0. So it's like an "and" over the features, but with a weighting that corresponds to the magnitude of agreement -- more agreement makes the output more positive.

$\endgroup$
5
  • $\begingroup$ Are you referring to this book ? If so I only find a mention of tanh on P.91, but it barely touches on the subject. Where were you able to find all the information in your answer? $\endgroup$
    – Ragnar
    Mar 3, 2016 at 21:06
  • 1
    $\begingroup$ I derived the information in my answer from my knowledge of linear algebra. $\endgroup$
    – Sycorax
    Mar 3, 2016 at 21:40
  • 1
    $\begingroup$ Sorry, I don't. The cool think about kernel methods is that we don't need to explicitly compute the feature map, though, and we can work with the kernel matrix directly. But this would be a great question to ask in its own right. $\endgroup$
    – Sycorax
    Mar 20, 2016 at 13:30
  • $\begingroup$ @Sycorax, in the quadratic equation you derive, isn't be = k_11 + k_22? You have b = k_11 - k_22 inside the radical. $\endgroup$
    – David R
    Jan 16, 2018 at 1:10
  • $\begingroup$ @DavidR No, it's just regrouping the terms. Hopefully this edit makes this clear. $\endgroup$
    – Sycorax
    Jan 16, 2018 at 16:07
1
$\begingroup$

There is a much simpler answer than Sycorax's.

A valid kernel must define a valid dot product, hence $\forall x, k(x, x) >= 0$ (with equality iff $x=0$).

Just take a negative value for $\nu$ and $x=0$.

$$k(x, x) = \tanh(\nu + 0) < 0$$

With positive values of $\nu$, there is no contradiction with just one point, but it doesn't mean you cannot find a contradiction for more points.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.