Kernel methods are used in machine learning to generalize linear techniques to nonlinear situations, especially SVMs, PCA, and GPs. Not to be confused with [kernel-smoothing], for kernel density estimation (KDE) and kernel regression.

learn more… | top users | synonyms

1
vote
0answers
22 views

How to make a customized kernel function

I have a sample set ${(\mathbf{x_i})}_{I=1}^N$, each $\mathbf{x_i}$ $\in R^d$ and $\mathbf{x_i}$ is a column vector with $d$ dimensions. Webpage gives an idea that a kernel function can be build ...
1
vote
0answers
16 views

Projection on weighted kernel PCA basis

I'm performing a sort of weighted kernel PCA, where the weights of samples can be negative. The weights of all samples are given by the diagonal weight matrix $D$. The data matrix is the $n \times d$ ...
2
votes
1answer
19 views

What will be the input value (i.e. $x$ and $x′$) of RBF kernel for a given dataset or data matrix $x$?

If $x$ is a data matrix or dataset then What will be the input value (i.e. $x$ and $x'$) of RBF kernel $K_r(x,x')=\exp(-\frac{\|x-x'\|^2}{r})$ ? I can understand $x$ is same as dataset or data matrix ...
0
votes
1answer
9 views

Kernel feature mapping: Derivation of polynomial kernel

The question is related to the derivation shown in section 3.1 Examples in the following lecture: http://people.eecs.berkeley.edu/~jordan/courses/281B-spring04/lectures/lec4.pdf I am confused about ...
1
vote
0answers
17 views

Is it possible to yield high-dimensional data from its low-dimensional point in KPCA?

With PCA, it is possible to reconstruct high-dimensional data from its low-dimensional point by $$ x_i' = Pb + \bar X $$ Where $\bar X$ is the mean of training set $X$, $P$ is the eigenvectors and $b$ ...
2
votes
1answer
30 views

What is the definition of a kernel on vertices or edges?

I am currently trying to perform clustering on a collection C of undirected and unlabeled graphs. I decided to use to a kernel on graphs to obtain the kernel matrix of C. Then I can derive the ...
1
vote
0answers
15 views

Reference request about feature maps in ML

Can someone kindly link to some recent papers on understanding feature maps in ML? It would help to get an idea of what are the recent issues there that people have been working on with regards to ...
0
votes
1answer
27 views

How to justify the usage of “Radial Basis Function” as a kernel for SVM [duplicate]

I ran SVM with several kernels on my data. RBS has the best performing results. The task is similar to the text classification. I wonder how I can explain why RBS is actually the best kernel for my ...
3
votes
1answer
40 views

Why must kernel functions be scalar products [duplicate]

I'm currently reading Bishop's Pattern Recognition and Machine Learning. In the chapter on kernel methods, he's very clear that kernels must be "valid", that is: be representable as scalar products in ...
0
votes
0answers
28 views

Does it make sense to do PCA before kernel regression?

I have a set of features extracted from the same samples and I'm learning a kernel ridge regression. Now, especially for feature fusion, reducing the number of features before combining them seems ...
1
vote
0answers
10 views

Is it needed to regularize in case you know your data is generated by a model of your model class?

Assume we have a dataset $X_{full}$ with labels $y_{full}$. We train a kernel ridge regression model on this data with the Gaussian kernel. This model is used to generate predictions on the whole ...
0
votes
1answer
52 views

How does gamma in SVM RBF kernel influence the accuracy?

I am working on a classification program using SVM RBF kernel. To find the best parameters C and gamma, I used grid search, and got the image below. What confuses me is that when gamma varies from 0.3 ...
0
votes
0answers
11 views

What happens to the RKHS of a Gaussian kernel if $\sigma$ is increased / decreased?

I'm especially interested in the following case: let's say we have a Gaussian kernel $K$ with bandwidth $\sigma$ and RKHS $\mathcal{H}$ and a set $H = \{ \forall h \in \mathcal{H} : ||h||_K \leq \...
0
votes
0answers
28 views

Gaussian kernel cosine distance

I have a dataset with clustering results from $K$=3 to 70 clusters for $n$ observations. My original dataset is an $n \times m$ matrix with $n$ rows (one for each observation) and $m$ features. I am ...
0
votes
0answers
8 views

How to find a similarity score of objects based on multiple nominal/ordinal/continuous variables

I am trying to solve a problem for finding similarity score between objects to create a similarity score matrix based on multiple nominal/ordinal/continuous variables for each object. Example of how ...
5
votes
2answers
71 views

Is Gradient Descent possible for kernelized SVMs (if so, why do people use Quadratic Programming)?

Why do people use Quadratic Programming techniques (such as SMO) when dealing with kernelized SVMs? What is wrong with Gradient Descent? Is it impossible to use with kernels or is it just too slow (...
1
vote
0answers
7 views

Circularity of Gaussian kernels and infinite dimensionality [closed]

I'd like to know more about this statement: Gaussian kernels are circular (which leads to the above-mentioned infinite dimensionality?) Questions: in image processing the issue of ...
0
votes
0answers
30 views

What is the problem with negative eigenvalue (in gram-matrix) in SVM?

This is probably very basic, but I still don't know the answer. I am working on homework in my course, and one of the questions is dealing with negative eigenvalue in gram matrix at SVM, can someone ...
0
votes
0answers
35 views

Creating a Radial basis function kernel matrix in matlab

I never used matlab, and I have this code about kernalized locality sensitive functions. I think that the following code is trying to create the kernalized matrix of a RBF kernel function: ...
3
votes
1answer
83 views

Learning functional analysis for studying kernels

I'm trying to learn more about kernel machine theory and I've discovered that I need to learn a lot of background math, and so I'm looking for some good resources for this. In particular: I've got ...
1
vote
0answers
15 views

Are VC dimensionality and dimension of kernel used in SVM related?

Are VC dimensionality and dimension of kernel used in SVM related to each other? or are they independent parameters in a classification process ?
0
votes
0answers
9 views

Representer Theorem: Is the RKHS $H$ the feature space or the estimator space?

I don't know how theoretical this page is, but I don't know a better place to ask this. It is more an intuitive question than a formal one. In reproducing kernel hilbert space theory we normally have ...
-1
votes
1answer
31 views

Why training error > 0 on SVM with RBF kernel

When using RBF kernel I think feature space is infinite dimensional space. With infinite dimensional features, I believe any training set can be classified. So I'm wondering why training error > 0 ...
1
vote
0answers
33 views

Understanding Kernel Ridge Regression and How It Works (and Implementing it in R)

I am trying to understand how KRR works for drug-protein-interaction and many aspects of it seem very confusing. Supposing I have a data set as follows of Drug-Protein interactions; values show how ...
0
votes
0answers
21 views

Parameter ranges for sigmoid and polynomial kernel

I would like to use a SVM classifier with a sigmoid and polynomial kernel. The sigmoid kernel has the following form: $K(u,v) = \tanh(\gamma * u'v + \text{coef}_{0})$ The polynomial kernel has the ...
1
vote
0answers
8 views

Are there kernel-based one-class sparse kernel-based outlier detection methods, e.g. one-class Relevance Vector Machine?

I have a commercial outlier detection problem in moderate dimension (8-25). We have a limited number of true positive tags and can roughly evaluate performance of various methods. So far, the 1-...
0
votes
0answers
15 views

Two ways of defining a squared kernel: are they equal?

Let's say I have a PDS kernel $K$, and a corresponding featuremap $\psi(x)$. I am interested in vectors in the RKHS / featurespace of $K$ with limited norm, the norm is limited by $\Lambda$. Now I ...
0
votes
1answer
26 views

What does the number 'Kernel Option' refer to in SVM?

I read that the performance of some kernel functions in SVM can change if we change the number known as kernel option. For example, this article states that kernel option of value 2 was used, http://...
0
votes
0answers
22 views

The feature space from Gaussian kernel is infinite-dimensional, are there countably or uncountably many basis?

My attempt: Let $x,y\in\mathbb{R}^d$. We already know the Fourier transform of a Gaussian function is a Gaussian function.If substituting $x-y$ for the variable after Fourier transform, we have $$ \...
0
votes
0answers
9 views

Difference between polynomial regression and polynomial kernel

A few answers on SO suggested that a polynomial transformation and a regularized regression can be used instead of a polynomial kernel regression. What's the difference between them? I thought ...
1
vote
1answer
25 views

Is a kernel space necessarily high dimensional?

It's very possible that I'm misunderstanding one or more terms here, so I will just try to explain what I understand (and why it doesn't make sense) :) Say I have an $N$x$D$ data matrix, i.e. $N$ ...
1
vote
0answers
24 views

Even property of Kernel in Non-parametric statistics

In Non-parametric statistics one requirement for the kernel is : $$ K(-u)=K(u) $$ for all values of $u$. This requirement ensures that the average of the corresponding distribution is equal to that of ...
0
votes
0answers
9 views

How to extract PCs in kpca

I am using RBF kernel in KPCA of Kernlab. What is the procedure for extracting the k pcs which explain the maximum variance. In KPCA, I learned that the PCA will be done in a higher dimensional space,...
0
votes
0answers
19 views

Spark big matrix computation

I am trying to see if Spark is fit for my problem. I would like to use kernel methods, as presented in Kernel method. For example, kernel k-means. One important feature of the kernel methods is that ...
4
votes
1answer
58 views

What's the physical meaning of the eigenvectors of the Gram/Kernel matrix?

If we have some centered dataset $X$ then the eigenvectors of $X^TX$ represent the principal components of the dataset, and their physical meaning is the directions that data follow in the original ...
0
votes
0answers
29 views

Kernel PCA increases dimensionality compared with PCA?

I was trying to use sklearn to perform kernel PCA with 28*28 = 784 dims data. At first I used PCA to reduce dimensionality and I chose to reduce to k dimensions where k could explain 95% of the ...
5
votes
1answer
62 views

Geometric Interpretation of Whether SVMs are performing well or not

I came across this research paper which contained this figure which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as ...
1
vote
2answers
91 views

Projecting to lower/higher-dimensional space for classification: dimensionality reduction vs kernel trick

Whilst learning about classification, I have seen two different arguments. One is that projecting the data to a lower-dimensional space, such as with PCA, makes the data more easily separable. The ...
2
votes
2answers
59 views

Understanding Kernel Functions for SVMs

I am learning about Support Vector Machines, and in particular, those with kernels for non-linear decision boundaries. I understand the concept of projecting the original data to a higher-dimensional ...
1
vote
1answer
38 views

What does a kernel function do in English

The kernel trick avoids the explicit mapping that is needed to get linear learning algorithms to learn a nonlinear function or decision boundary. For all and in the input space , certain functions can ...
1
vote
1answer
168 views

On the properties of Hyperbolic Tangent Kernel

I've read from various sources that Hyperbolic Tangent kernels are not positive semi-definite and thus are not actually a valid kernel. Does this mean they are misnomer? Furthermore, if they are ...
1
vote
1answer
38 views

How are Hyperplane Heatmaps created and how should they be interpreted?

For nonlinear data, when we are using Support Vector Machines, we can use kernels such as Gaussian RBF, Polynomial, etc to achieve linearity in a different (potentially unknown to us) feature space ...
2
votes
1answer
22 views

Proving the validity of a kernel

How can we prove that the following is a valid kernel? Let $\phi$ be any function on R X R. Define: $K(x,y) = \int{\phi(x,z)\phi(y,z)dz}$ We want to show that $K$ is a valid kernel.
0
votes
1answer
50 views

Why can kernel PCA with Gaussian kernel separate half-moon shapes and concentric circles but not Swiss Roll?

According to this website, kernel PCA with RBF (Gaussian) kernel can separate half-moon shapes and concentric circles effectively but not Swiss Roll shapes (in 3-D). I don't understand why it doesn't ...
0
votes
0answers
17 views

SVM kernels combination

Let's suppose we have classification problem with two classes. $$X^l = (x_i, y_i)_{i=1}^l, Y=\{+1, -1\}\;\; y:X\rightarrow Y$$ $$x_i \in R^n, y_i = y(x_i),\;\; i = 1\dots n$$ One of the most spread ...
4
votes
1answer
63 views

Machine Learning SVM

If one trains a model using a SVM from kernel data, the resultant trained model contains support vectors. Now consider the case of training a new model using the old data already present plus a small ...
0
votes
0answers
34 views

Rules for choosing how much training data one needs to learn a Radial Basis Function (RBF) model?

I was trying to understand how much data I would need compared to the number of parameters (and to have good generalization) when I train a radial basis function (RBF) network on a regression task ...
1
vote
2answers
55 views

Corresponding RKHS of Common Kernels

A kernel, $k(x_1, x_2)$, has the interesting property that it may be represented as the dot product in a reproducing kernel hilbert space (RKHS), $\phi(x_0)\phi(x_1)$. I know that for the gaussian ...
1
vote
1answer
69 views

What happens if you square an RBF kernel function?

Let's say we use a kernel regularization algorithm such as ridge regression to minimize some loss in an RBF kernel: $$\min_{h \in H} \frac{1}{n} \sum_i (h(x_i) - y(x_i))^2 + ||h||^2_K$$ We get some $...
1
vote
0answers
39 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?