Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

learn more… | top users | synonyms

1
vote
0answers
5 views

Are there kernel-based one-class sparse kernel-based outlier detection methods, e.g. one-class Relevance Vector Machine?

I have a commercial outlier detection problem in moderate dimension (8-25). We have a limited number of true positive tags and can roughly evaluate performance of various methods. So far, the ...
0
votes
0answers
15 views

Two ways of defining a squared kernel: are they equal?

Let's say I have a PDS kernel $K$, and a corresponding featuremap $\psi(x)$. I am interested in vectors in the RKHS / featurespace of $K$ with limited norm, the norm is limited by $\Lambda$. Now I ...
0
votes
1answer
22 views

What does the number 'Kernel Option' refer to in SVM?

I read that the performance of some kernel functions in SVM can change if we change the number known as kernel option. For example, this article states that kernel option of value 2 was used, ...
0
votes
0answers
22 views

The feature space from Gaussian kernel is infinite-dimensional, are there countably or uncountably many basis?

My attempt: Let $x,y\in\mathbb{R}^d$. We already know the Fourier transform of a Gaussian function is a Gaussian function.If substituting $x-y$ for the variable after Fourier transform, we have $$ ...
0
votes
0answers
7 views

Difference between polynomial regression and polynomial kernel

A few answers on SO suggested that a polynomial transformation and a regularized regression can be used instead of a polynomial kernel regression. What's the difference between them? I thought ...
1
vote
1answer
24 views

Is a kernel space necessarily high dimensional?

It's very possible that I'm misunderstanding one or more terms here, so I will just try to explain what I understand (and why it doesn't make sense) :) Say I have an $N$x$D$ data matrix, i.e. $N$ ...
1
vote
0answers
24 views

Even property of Kernel in Non-parametric statistics

In Non-parametric statistics one requirement for the kernel is : $$ K(-u)=K(u) $$ for all values of $u$. This requirement ensures that the average of the corresponding distribution is equal to that of ...
0
votes
0answers
7 views

How to extract PCs in kpca

I am using RBF kernel in KPCA of Kernlab. What is the procedure for extracting the k pcs which explain the maximum variance. In KPCA, I learned that the PCA will be done in a higher dimensional ...
0
votes
0answers
15 views

Spark big matrix computation

I am trying to see if Spark is fit for my problem. I would like to use kernel methods, as presented in Kernel method. For example, kernel k-means. One important feature of the kernel methods is that ...
4
votes
1answer
48 views

What's the physical meaning of the eigenvectors of the Gram/Kernel matrix?

If we have some centered dataset $X$ then the eigenvectors of $X^TX$ represent the principal components of the dataset, and their physical meaning is the directions that data follow in the original ...
0
votes
0answers
26 views

Kernel PCA increases dimensionality compared with PCA?

I was trying to use sklearn to perform kernel PCA with 28*28 = 784 dims data. At first I used PCA to reduce dimensionality and I chose to reduce to k dimensions where k could explain 95% of the ...
5
votes
1answer
60 views

Geometric Interpretation of Whether SVMs are performing well or not

I came across this research paper which contained this figure which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as ...
1
vote
2answers
67 views

Projecting to lower/higher-dimensional space for classification: dimensionality reduction vs kernel trick

Whilst learning about classification, I have seen two different arguments. One is that projecting the data to a lower-dimensional space, such as with PCA, makes the data more easily separable. The ...
2
votes
2answers
49 views

Understanding Kernel Functions for SVMs

I am learning about Support Vector Machines, and in particular, those with kernels for non-linear decision boundaries. I understand the concept of projecting the original data to a higher-dimensional ...
1
vote
1answer
35 views

What does a kernel function do in English

The kernel trick avoids the explicit mapping that is needed to get linear learning algorithms to learn a nonlinear function or decision boundary. For all and in the input space , certain functions ...
1
vote
1answer
140 views

On the properties of Hyperbolic Tangent Kernel

I've read from various sources that Hyperbolic Tangent kernels are not positive semi-definite and thus are not actually a valid kernel. Does this mean they are misnomer? Furthermore, if they are ...
1
vote
1answer
36 views

How are Hyperplane Heatmaps created and how should they be interpreted?

For nonlinear data, when we are using Support Vector Machines, we can use kernels such as Gaussian RBF, Polynomial, etc to achieve linearity in a different (potentially unknown to us) feature space ...
2
votes
1answer
21 views

Proving the validity of a kernel

How can we prove that the following is a valid kernel? Let $\phi$ be any function on R X R. Define: $K(x,y) = \int{\phi(x,z)\phi(y,z)dz}$ We want to show that $K$ is a valid kernel.
0
votes
1answer
40 views

Why can kernel PCA with Gaussian kernel separate half-moon shapes and concentric circles but not Swiss Roll?

According to this website, kernel PCA with RBF (Gaussian) kernel can separate half-moon shapes and concentric circles effectively but not Swiss Roll shapes (in 3-D). I don't understand why it doesn't ...
0
votes
0answers
17 views

SVM kernels combination

Let's suppose we have classification problem with two classes. $$X^l = (x_i, y_i)_{i=1}^l, Y=\{+1, -1\}\;\; y:X\rightarrow Y$$ $$x_i \in R^n, y_i = y(x_i),\;\; i = 1\dots n$$ One of the most spread ...
4
votes
1answer
60 views

Machine Learning SVM

If one trains a model using a SVM from kernel data, the resultant trained model contains support vectors. Now consider the case of training a new model using the old data already present plus a small ...
0
votes
0answers
28 views

Rules for choosing how much training data one needs to learn a Radial Basis Function (RBF) model?

I was trying to understand how much data I would need compared to the number of parameters (and to have good generalization) when I train a radial basis function (RBF) network on a regression task ...
1
vote
2answers
41 views

Corresponding RKHS of Common Kernels

A kernel, $k(x_1, x_2)$, has the interesting property that it may be represented as the dot product in a reproducing kernel hilbert space (RKHS), $\phi(x_0)\phi(x_1)$. I know that for the gaussian ...
1
vote
1answer
56 views

What happens if you square an RBF kernel function?

Let's say we use a kernel regularization algorithm such as ridge regression to minimize some loss in an RBF kernel: $$\min_{h \in H} \frac{1}{n} \sum_i (h(x_i) - y(x_i))^2 + ||h||^2_K$$ We get some ...
1
vote
0answers
26 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
24 views

has anyone tried to use spectrum kernel [duplicate]

Can any one explain me how to use string kernels to quantify the similarity between short texts? thank you. regards
1
vote
1answer
108 views

In Convolutional Neural Networks (CNN), how we can decide number of kernels between input and hidden layer?

I have $32\times32$ input image and $5\times5$ convolution. So in the first hidden layer, the feature map size will be $28\times28$. At this link we can see in C1, the number of feature maps is 4 but ...
0
votes
0answers
16 views

After running kernel PCA on the training data, how to apply it to a new data point? [duplicate]

My data set has a training set of 1000 input with 6 features (data set size is 1000*6). I applied kernel PCA to the data set and reduced the number of features to 3. It means that the dimension of the ...
1
vote
0answers
24 views

Gaussian Process: Parameter of Kernel function

I am quite new to kernel method, I am trying to estimate $y'$ corresponding to $x'$, given [x, y] data. I am using Gaussian Method for analysis with Kernel function: $k(x_1,x_2 ) =p_1\exp\{-p_2(x_1 ...
0
votes
0answers
16 views

Choosing orientation for kernel principal component analysis

I have a matrix with orientation [1500x93]. The 1500 corresponds to 1500 time samples and the 93 corresponds to the X,Y and Z coordinates for 31 markers. I am performing principal component analysis ...
0
votes
0answers
47 views

How to compare PCA with KPCA for dimension reduction?

Both linear principal component analysis (PCA) and kernel principal component analysis (KPCA) are unsupervised dimension reduction methods. I have a dataset with $4000$ training samples and $40000$ ...
1
vote
1answer
43 views

linear kernel SVM

The linear kernel is defined as: $K(x1,x2)=\langle x1,x2\rangle$. I can see that all that this kernel does is to calculate the dot product in the original space of the data. Why is this kernel then ...
0
votes
0answers
45 views

Mercer's condition

I am having a very hard time understanding Mercer kernel. If any sequence of data points $x_1, ... , x_n \in R^d$ and coefficients $c_1, ... , c_n \in R$, satisifies the inequality $\sum^n_{i=1} ...
0
votes
0answers
19 views

SVM - relevance of linear kernel

The linear kernel is of the form K(x1,x2)=$<x1,x2>$. I understand that kernel functions help us compute a dot product in some high dimensional space. In the case of the linear kernel, I see that ...
1
vote
0answers
31 views

Stochastic Differential Equation Interpretation of Squared Exponential Kernel

As far as I understand, many Gaussian Processes can be either described by their corresponding mean and kernel functions or by a stochastic differential equation (SDE). For my purposes it is ...
0
votes
0answers
28 views

Why is the squared exponential kernel so popular?

Be it SVM or GPR is seems like, besides the linear kernel, in the kernel machines community the squared exponential kernel $$k(x,x')=\sigma^2\exp\left((x-x')^2/l\right)$$ with $\sigma>0$ and ...
0
votes
1answer
50 views

Eigenfunctions and eigenvalues of the exponential kernel

What are the eigenfunctions and the eigenvalues of the exponential kernel? The exponential kernel is defined as $$k(x,x')=\sigma^2\exp\left(\frac{||x-x'||}{l}\right)$$ where both $\sigma>0$ and ...
0
votes
0answers
11 views

Prove that $k(x,x') = k_a(x_a, x'_a) + k_b(x_b, x'_b)$ where $x = (x_a,x_b)$ is a kernel

Prove that $k(x,x') = k_a(x_a, x'_a) + k_b(x_b, x'_b)$ where $x = (x_a,x_b)$ is a kernel. I can prove that $k(x,x') = k_1(x,x') + k_2(x,x')$ is a kernel but I cannot see how this can be used to solve ...
0
votes
1answer
53 views

PCA vs. Spectral Clustering with Linear Kernel

Consider a feature vector matrix $X := [x_1 x_2 \dots x_d] \in \mathbb {R}^{n\times d} $ that I hope to use as part of some supervised learning procedure, say, regression. Suppose that also, $d \gg n ...
1
vote
0answers
35 views

Can a SVM with polynomial kernel lead to overfitting?

I'm currently searching for the best set of parameters for a Polynomial Kernel with a grid search. I would like to know if using a high value for Polynomial order (10, 100 or 1000) can lead to ...
1
vote
0answers
28 views

SVM - support vectors on the wrong side of the margin for polynomial kernel

In soft-margin SVM, the constraint in soft-margin (compared to hard-margin) is $0≤α_i≤C$ for some positive constant C. We say that: If $α_i=C$, then the corresponding $x_i$ is on the wrong side of ...
0
votes
0answers
20 views

Rademacher complexity of SVM with kernel in terms of whole Kernel Matrix

http://www.cs.nyu.edu/~mohri/mls/lecture_5.pdf In slide no. 18 here, it is shown that Rademacher complexity of SVM with kernel can be written in terms of trace of the matrix. Are there any other ...
1
vote
1answer
24 views

Use RBF kernel with logistic regression?

There are some resources online (e.g. this one) on logistic regression with polynomial kernels, such as $$h_\theta(x)=logistic(\theta_0 + \theta_1x1+ \theta_3x_1^2 + \theta_4x_2^2)$$ I'm wondering ...
4
votes
2answers
108 views

Is there any kernel defined for simplex domain (i.e. probability vector)?

I am wondering if there is any kernel function that is specifically designed for simplex domain. By "simplex domain," I mean a set whose elements are probability vectors. For example, 3-D simplex may ...
1
vote
0answers
33 views

What does the constraint on the signed support vectors in SVMs signify?

What does the constraint $\sum_i \alpha_i y_i = 0$ on the support vectors signify? Does it mean a data set cannot have only one support vector? Can all the support vectors of a data set after ...
2
votes
1answer
34 views

What does prime mean in the notation $k(x,x')$ in the context of kernels?

I admit it's a very specific question, but what is meant by the notation $'$ in Bishop's book Pattern Recognition and Machine Learning? I'd always thought it's a transpose, but this marked by a ...
3
votes
1answer
351 views

Implicit feature space of Power Kernel

For the polynomial kernel, $K(x,y) = (x^Ty+c)^d$, the implicit feature space $\phi$ for which $K(x,y) = \phi(x)^T \phi(y)$ is of finite dimension and well known [1][2]. It is also well known that the ...
3
votes
1answer
102 views

How to prove or disprove this function is valid kernel?

I have the following function $$ K(x, y) = \begin {cases} 1, & if ||x - y||_2 \le 1 \\ 0, & otherwise \end{cases} $$ I'd like to prove (or disprove) that it's a valid kernel function. In ...
1
vote
0answers
200 views

Implementing kernel ridge regression

I want to implement kernel ridge regression in R. My problem is that I can't figure out how to generate the kernel values and I do not know how to use them for the ridge regression. Before going to ...
0
votes
0answers
31 views

VC dimension of SVM with polynomial kernel in $\mathbb{R}^d$?

Following along the lines of the question here: VC dimension of SVM with polynomial kernel in $\mathbb{R^{2}}$ What is the equation for VC-dim of an SVM with a 2nd-degree polynomial kernel ...