All Questions

Tagged with
Filter by
Sorted by
Tagged with
0
votes
1answer
29 views

Kernel PCA: Find most important variables for each PC

To find the most important variable for each Principal Component is easy with PCA: With data->X and variables->variable_names ...
0
votes
0answers
18 views

Denoising and pre-images in Kernel PCA

In "Pattern Recognition and Machine Learning" by Bishop, the following problem about Kernel PCA is laid out : In linear PCA, we can approximate data points by projecting them onto the $L < D$-...
0
votes
0answers
26 views

Transforming the Kernel principal components to original space

According to my understanding, we obtain the kernel/gram matrix eigenvectors/values in kernel PCA. We can use the kernel matrix for transforming the data however is there a way to transform those ...
0
votes
0answers
67 views

If I center my kernel does it no longer remain positive semidefinite?? If so why is it being used in algorithms like kernel pca?

If I center my kernel then can it still be used in operations where a positive semi-definite kernel is required such as SVM and ridge regression? I am centering my kernel as follows: $$K_c(\mathbf{t}...
2
votes
0answers
61 views

Kernel function for use in Kernel-PCA given a known piecewise linear true data generating process

If I know that a multivariate dataset has a piecewise-linear data generating process with known knots (or breakpoints), then what is the appropriate kernel function to use in Kernel-PCA? For example, ...
3
votes
0answers
94 views

Understanding kernel PCA when the target space is infinite-dimensional

The PCA optimization problem is known as $$ \max_{U \in \mathbb{R}^{d\times r}, U^TU = I} tr(U^T\Sigma U), $$ where $\Sigma$ is a covariance matrix of a dataset $\{x_1,\dots,x_n\} \subset \mathbb{R}^d$...
0
votes
1answer
113 views

SVM: Maximal number of components kernel PCA versus linear PCA

I'm comparing the Support Vector Machines (SVM) formulation of linear PCA with kernel PCA. I know that in linear PCA, the maximum number of principal components is equal to the dimension of the input ...
1
vote
1answer
215 views

How to understand effect of RBF kernel for kernel PCA

I understand the math in kernel PCA and with RBF kernel, and I also understand that the RBF kernel map the data into a infinite dimensional space. I know that for SVM, mapping the data into a higher ...
1
vote
0answers
140 views

The gamma parameter (or kernel width) for RBF Gaussian kernel in kernel PCA

Is there any general way or rule of thumb of how to determine the kernel width for KPCA?
0
votes
1answer
180 views

Relationship between PCA and KPCA

The relation between PCA and KPCA seems somewhat confusing. Basically, the Kernel variant of PCA can be described as constructing the normalized kernel matrix of the data $n \times n$ (n is the number ...
0
votes
0answers
92 views

Why shouldn't we scale data before entering into KPCA?

In Kernel Principal Component Analysis (KPCA), data comes in as a $n\times d$ matrix $X$ where $n$ is the number of observations and $d$ is the number of features. The process has been explained in ...
1
vote
1answer
357 views

KernelPCA from sklearn doesn't return original data

If I do a transform and then an inverse_transform using PCA, I of course get the original data back. If I do the same for KernelPCA, I don't. Is this a property of kernelPCA or a shortcoming of the ...
3
votes
1answer
441 views

Principal Component Analysis on Graphs

How can somebody apply PCA on a set of graphs? Is it possible to define a meaningful graph kernel for my problem, and then follow the typical procedure on the derived matrix of pairwise distances (...
3
votes
0answers
220 views

Normalization of eigenvector $\alpha$ of kernel PCA by eigenvalue $\lambda$ [duplicate]

Could someone please help me understand why when we project a new sample in kernel PCA to an eigenvector $\alpha$, we normalize it by dividing the eigenvector $\alpha$ with its eigenvalue $\lambda$? ...
2
votes
0answers
387 views

Intuition of KernelPCA

I'm dealing currently with kernels and kernel PCA. For this purpose I've been reading a few papers on these topics. In this context I've been reading the paper "Kernel Principal Component Analysis" by ...
1
vote
0answers
90 views

Major and minor principal axes using kernel PCA on a 2D dataset

I have these two data sets: ...
2
votes
0answers
156 views

About the effect of the width of the Gaussian kernel

If one does say "Kernel Ridge Regression" or "Kernel PCA" using the Gaussian kernel then do we know how the choice of the width of the Gaussian kernel affects the quality of the answer? Like does in ...
3
votes
1answer
421 views

Eigenvalue equation for kernel PCA

In Nonlinear component analysis as a kernel eigenvalue problem, Schölkopf et al start by describing PCA. Given a set of data instances $x_1, \dots, x_M$, with $x_k \in \mathbb{R}^N, k=1,\dots,M$, and ...
1
vote
0answers
107 views

Projection on weighted kernel PCA basis

I'm performing a sort of weighted kernel PCA, where the weights of samples can be negative. The weights of all samples are given by the diagonal weight matrix $D$. The data matrix is the $n \times d$ ...
2
votes
0answers
32 views

Is it possible to yield high-dimensional data from its low-dimensional point in KPCA?

With PCA, it is possible to reconstruct high-dimensional data from its low-dimensional point by $$ x_i' = Pb + \bar X $$ Where $\bar X$ is the mean of training set $X$, $P$ is the eigenvectors and $b$ ...
1
vote
0answers
24 views

Reference request about feature maps in ML

Can someone kindly link to some recent papers on understanding feature maps in ML? It would help to get an idea of what are the recent issues there that people have been working on with regards to ...
0
votes
0answers
93 views

Does it make sense to do PCA before kernel regression?

I have a set of features extracted from the same samples and I'm learning a kernel ridge regression. Now, especially for feature fusion, reducing the number of features before combining them seems ...
5
votes
1answer
2k views

What's the physical meaning of the eigenvectors of the Gram/Kernel matrix?

If we have some centered dataset $X$ then the eigenvectors of $X^TX$ represent the principal components of the dataset, and their physical meaning is the directions that data follow in the original ...
1
vote
1answer
651 views

Kernel PCA increases dimensionality compared with PCA?

I was trying to use sklearn to perform kernel PCA with 28*28 = 784 dims data. At first I used PCA to reduce dimensionality and I chose to reduce to k dimensions where k could explain 95% of the ...
4
votes
2answers
3k views

Projecting to lower/higher-dimensional space for classification: dimensionality reduction vs kernel trick

Whilst learning about classification, I have seen two different arguments. One is that projecting the data to a lower-dimensional space, such as with PCA, makes the data more easily separable. The ...
1
vote
1answer
507 views

Why can kernel PCA with Gaussian kernel separate half-moon shapes and concentric circles but not Swiss Roll?

According to this website, kernel PCA with RBF (Gaussian) kernel can separate half-moon shapes and concentric circles effectively but not Swiss Roll shapes (in 3-D). I don't understand why it doesn't ...
0
votes
1answer
261 views

How to compare PCA with KPCA for dimension reduction?

Both linear principal component analysis (PCA) and kernel principal component analysis (KPCA) are unsupervised dimension reduction methods. I have a dataset with $4000$ training samples and $40000$ ...
1
vote
1answer
598 views

PCA vs. Spectral Clustering with Linear Kernel

Consider a feature vector matrix $X := [x_1 x_2 \dots x_d] \in \mathbb {R}^{n\times d} $ that I hope to use as part of some supervised learning procedure, say, regression. Suppose that also, $d \gg n $...
4
votes
0answers
220 views

Kernel PCA vs principal curve analysis

Both principal curve analysis and kernel PCA provide the ability to find nonlinear PCA. Kernel PCA does this by finding principal components in a higher dimensional space. Principal curve analysis is ...
2
votes
1answer
504 views

Kernel PCA for feature selection for various machine learning algorithms [duplicate]

I would like to forecast stock index returns with SVM, k-NN, and Neural Networks. In advance I want to select my inputs via kernel PCA (kPCA). Everything is performed in R. For the KPCA I use ...
2
votes
1answer
1k views

A problem with kernel-PCA implementation

Linear PCA and kPCA with linear kernel should produce exactly the same results ( good explanation is in this post ). As I am learning to use PCA family methods I try to write my own functions ...
3
votes
0answers
313 views

Generalization error of PCA and kernel PCA

I've been recently reading Shawe-Taylor et al. 2005, On the Eigenspectrum of the Gram Matrix and the Generalization Error of Kernel PCA, where the authors analyze the squared residual of kernel ...
1
vote
2answers
2k views

Kernel PCA and classification

I need to perform kernel PCA on the colon-­‐cancer dataset and then I need to plot number of principal components vs classification accuracy with PCA data. For the first part I am ...
11
votes
1answer
4k views

How to choose a kernel for kernel PCA?

What are the ways to choose what kernel would result in good data separation in the final data output by kernel PCA (principal component analysis), and what are the ways to optimize parameters of the ...
7
votes
1answer
3k views

What exactly is the procedure to compute principal components in kernel PCA?

In kernel PCA (principal component analysis) you first choose a desired kernel, use it to find your $K$ matrix, center the feature space via the $K$ matrix, find its eigenvalues and eigenvectors, then ...
1
vote
1answer
107 views

Integrating length for input-space feature PC projections in kernel PCA

I read a paper detailing the algebraic process of kernel PCA. I have question though: the paper details the projection of new points onto the new eigenvectors in the feature space, but what if I want ...
64
votes
4answers
36k views

What makes the Gaussian kernel so magical for PCA, and also in general?

I was reading about kernel PCA (1, 2, 3) with Gaussian and polynomial kernels. How does the Gaussian kernel separate seemingly any sort of nonlinear data exceptionally well? Please give an intuitive ...
2
votes
1answer
157 views

How to fit a single quadratic term to a regression

I have a high dimensional multivariate model and am fitting linear weights to each of the $N$ free variables using a classic stable SVD matrix solver. This works. I want to improve the fit by using a ...
4
votes
1answer
6k views

How to project a new vector onto the PC space using kernel PCA?

Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space. Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and ...
1
vote
0answers
75 views

Non-decaying eigenvalues in Kernel PCA with small kernel width

I noticed that when I use a small width kernel (RBF) with PCA, I get my desired result (clustering in this case), but I do not get a decay in the eigenvalues (they stay about the same value). Is that ...
1
vote
1answer
759 views

Which PCA (or kernel PCA) basis better describes a single test sample?

I have two PCA bases obtained by decomposition of two groups of training data. I also have some samples of test data. How can I decide which PCA basis fits better each test sample? I tried to ...
7
votes
1answer
8k views

How to apply a Gaussian radial basis function kernel PCA to nonlinear data?

I have an assignment to implement a Gaussian radial basis function-kernel principal component analysis (RBF-kernel PCA) and have some challenges here. It would be great if someone could point me to ...
5
votes
1answer
422 views

Are eigenvectors obtained in Kernel PCA orthogonal?

As Kernel PCA is the same as PCA in higher dimension space, shouldn't the eigenvectors obtained be orthogonal? Suppose, I have $n$ data points and let $a$ and $b$ be two eigenvectors of covariance ...
17
votes
3answers
8k views

Is Kernel PCA with linear kernel equivalent to standard PCA?

If in kernel PCA I choose a linear kernel $K(\mathbf{x},\mathbf{y}) = \mathbf x^\top \mathbf y$, is the result going to be different from the ordinary linear PCA? Are the solutions fundamentally ...
14
votes
1answer
15k views

What are the advantages of kernel PCA over standard PCA?

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to ...
3
votes
2answers
2k views

How to select a number of components to retain in kernel PCA?

I'm using kpca function from kernlab and try to get the proportion of variance explained by each component as in standard PCA. I ...
1
vote
1answer
461 views

Can one use eigenvalues to choose a number of components to retain in kernel PCA?

When using Kernel PCA for dimensionality reduction, is there any simple criterion which can be used to determine the number of components to use? I am using Kernel PCA with linear kernel, which would ...
2
votes
1answer
550 views

Kernel PCA with an SVD algo

Suppose that I have a great algo for calculating the SVD and I want to do Kernel PCA. It is possible to first apply the Kernel function to my data and then run the SVD algo on the transformed data?
2
votes
1answer
651 views

How to transform this dataset to make classes linearly separable?

I have this data set: And I want to transform the data (with a RBF kernel?) in order to be able to do a simple linear ridge-classifier. I know I can do more or less the same thing using a kernel ...
0
votes
1answer
528 views

Non-Orthogonality in PCA? [duplicate]

i) What is the main role of "only" trying to find orthogonal components in PCA? I can understand, that we would not want a zero-solution as well as find directions that are orthogonal in order to ...