Hot answers tagged kernel
11
For a univariate KDE, you are better off using something other than Silverman's rule which is based on a normal approximation. One excellent approach is the Sheather-Jones method, easily implemented in R; for example,
plot(density(precip, bw="SJ"))
The situation for multivariate KDE is not so well studied, and the tools are not so mature. Rather than a ...
9
It depends a little bit on what your end goal is.
Quick and dirty hack for real-time demonstrations
Using Sys.sleep(seconds) in a loop where seconds indicates the number of seconds between frames is a viable option. You'll need to set the xlim and ylim parameters in your call to plot to make things behave as expected.
Here's some simple demonstration ...
9
Zen used method 1. Here is method 2: Map $x$ to a spherically symmetric Gaussian distribution centered at $x$ in the Hilbert space $L^2$. The standard deviation and a constant factor have to be tweaked for this to work exactly. For example, in one dimension,
$$ \int_{-\infty}^\infty \frac{\exp[-(x-z)^2/(2\sigma^2)]}{\sqrt{2 \pi} \sigma} ...
8
For x,y on S, certain functions K(x,y) can be expressed as an inner product (in usually a different space). K is often referred to as a kernel or a kernel function. The word kernel is used in different ways throughout mathematics, but this is the most common usage in machine learning.
The kernel trick is a way of mapping observations from a general set S ...
8
The Moore-Aronszajn theorem guarantees that a symmetric positive definite kernel is associated to a unique reproducing kernel Hilbert space. (Note that while the RKHS is unique, the mapping itself is not.)
Therefore, your question can be answered by exhibiting an infinite-dimensional RKHS corresponding to the Gaussian kernel (or RBF). You can find an ...
7
To shamelessly quote the Stata manual entry for kdensity:
The optimal width is the width that would minimize the mean integrated squared error if the data were Gaussian and a Gaussian kernel were used, so it is not optimal in any global sense. In fact, for multimodal and highly skewed densities, this width is usually too wide and oversmooths the density ...
7
The optimal bandwidth for derivative estimation will be different from the bandwidth for density estimation. In general, every feature of a density has its own optimal bandwidth selector.
If your objective is to minimize mean integrated squared error (which is the usual criterion) there is nothing subjective about it. It is a matter of deriving the value ...
6
Basically anything what is not separable with a line (ok, hyperplane), for instance 2D data like this:
kernel trick will effectively project this situation into a (higher-dim) space in which linear separation is possible; see this movie for an effect of a gaussian kernel on similar data.
Look for a kernel argument in your svm function ;-) Note that using a ...
6
One way to go is to use the excellent animation package by Yihui Xie. I uploaded a very simple example to my public dropbox account: densityplot (I will remove this example in 3 days). Is this what you are looking for?
The animation was created using the following R code:
library(animation)
density.ani <- function(){
i <- 1
d <- ...
6
I think the main problem is to get the pairwise distances efficiently. Once you have that the rest is element wise.
To do this, you probably want to use scipy. The function scipy.spatial.distance.pdist does what you need, and scipy.spatial.distance.squareform will possibly ease your life.
So if you want the kernel matrix you do
from scipy.spatial.distance ...
6
I will use method 1. Check Douglas Zare answer for a proof using method 2.
I will prove the case when $x,y$ are real numbers, so $k(x,y)=\exp(-(x-y)^2/2\sigma^2)$. The general case follows mutatis mutandis from the same argument, and is worth doing.
Without loss of generality, suppose that $\sigma^2=1$.
Write $k(x,y)=h(x-y)$, where ...
5
Do your analysis with several different kernels. Make sure you cross-validate. Choose the kernel that performs the best during cross-validation and fit it to your whole dataset.
/edit: Here is some example code in R, for a classification SVM:
#Use a support vector machine to predict iris species
library(caret)
library(caTools)
#Choose x and y
x <- ...
5
No, this kernel doesn't give a positive (semi-) definite Gram matrix, so it is not a valid kernel (the first randomly generated Gram matrix $\matrix{K}$ and vector $\vec{v}$ I tried gave $\vec{v}'\matrix{K}\vec{v} < 0$ so $\matrix{K}$ isn't positive semi-definite. If you infer from that that I am an engineer - you are correct! ;o).
ISTR that Kernel ...
5
I've written a couple ;o)
G. C. Cawley and N. L. C. Talbot, Efficient approximate leave-one-out cross-validation for kernel logistic regression, Machine Learning, vol, 71, no. 2-3, pp. 243--264, June 2008.
Which gives a reasonable method for choosing kernel and regularisation parameters and an empirical evaluation
G. C. Cawley, G. J. Janacek and N. L. C. ...
5
The answering question is "why do you model your data as a sample from a distribution?" If you want to learn something about the phenomenon behind your data, like when improving a scientific theory or testing a scientific hypothesis, using a non-parametric kernel estimator does not tell you much more than the data istself. While a parameterised model can ...
5
In a machine learning context (i.e. "kernel methods"), the key requirement for a kernel is that it must be symmetric and positive-definite, that is, if $K$ is a kernel matrix, then for any (column) vector $x$ of the appropriate length, $x^{T}Kx$ must be a positive real number. This restriction is in place mostly due to requirements of optimization processes ...
4
Two further references from B. Schölkopf:
Schölkopf, B. and Smola, A.J. (2002). Learning with kernels. The MIT Press.
Schölkopf, B., Tsuda, K., and Vert, J.-P. (2004). Kernel methods in computational biology. The MIT Press.
and a website dedicated to kernel machines.
4
You can use conditional kernel density estimation to obtain the density of sales at time $t+h$ conditional on the values of sales at times $t, t-1, t-2, \dots$ This gives you a density forecast rather than a point forecast. The problem is that the conditioning is difficult in a density setting when the number of conditioning variables is more than 2. See ...
4
I asked a similar question a few months ago. Rob Hyndman provided an excellent answer that recommends the Sheather-Jones method.
One addition point. In R, for the density function, you set the bandwidth explicitly via the bw argument. However, I often find that the adjust argument is more helpful. The adjust argument scales the value of the bandwidth. So ...
4
My answer: I assume $(X,Y)$ is abs cont, i.e. has a density $f_{(X,Y)}$ with respect to Lebesgue measure in $\mathbb{R}^2$. Everything can be done by using parameter "weights" of function density in R, i.e. with a call to
$$\text{density}\left ((Y_1,\dots,Y_n),\; \text{ weights}=(e^{|x-X_i|^2/h^2})_{i=1,\dots,n}\right )$$
to estimate ...
4
Here is another approach:
library(TeachingDemos)
d <- c(1,2,3,4)
tmpfun <- function(width=1, kernel='gaussian'){
plot(density(d, width=width, kernel=kernel))
}
tmplst <- list( width=list('slider', init=1, from=.5, to=5, resolution=.1),
kernel=list('radiobuttons', init='gaussian', values=c('gaussian',
...
4
You are right about the three issues you raise, and your interpretation is exactly right.
People have looked at other directions to build kernels from probabilistic models:
Moreno et al. propose Kullback-Liebler although when this satisfies Mercer's conditions was not well understood when I looked at this problem back when I read it.
Jebara et al. ...
4
Grid search is a sensible procedure as @JohnSmith suggests, however it is not the only stable technique. I generally use the Nelder-Mead simplex algortihm, which I have found to be very reliable and more efficient than grid search as less time is spent investigating areas of hyper-parameter space that give poor models. If you are a MATLAB user, you can get ...
4
If you look at the optimization problem that SVM solves:
$\min_{\mathbf{w},\mathbf{\xi}, b } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^n \xi_i \right\}$
s.t.
$y_i(\mathbf{w}\cdot\mathbf{x_i} - b) \ge 1 - \xi_i, ~~~~\xi_i \ge 0,$
for all $ i=1,\dots n$
the support vectors are those $x_i$ where the corresponding $\xi_i \gt 0$. In other words, ...
4
You can think of the Kernel Density Estimation as a smoothed histogram. Histograms are limited by the fact that they are inherently discrete (via bins) and are thus more appropriate for displaying data on discrete variables and can be very sensitive to bin size.
What you are actually doing with the Kernel Density Estimation is estimating the probability ...
3
The polynomial kernel $K(x,y) = (x \cdot y + 1)^d$ is easily represented in terms of monomials. The degree $d$ is the maximum degree of the polynomial computed by the kernel and therefore also the maximum degree of any contained monomial.
The problem of determining the number of monomials of degree exactly $d$ in $p$ input variables is the same as the ...
3
Just for the sake of completeness, if you need this for a class demonstration, I would also mention the manipulate package which comes with RStudio. Note that this package is dependent on RStudio interface, so it won't work outside of it.
manipulate is quite cool because it allows to quickly create some sliders to manipulate any element in the plot. This ...
3
The conditional mean is defined by:
$$E(Y|X)\equiv\int y f(y|x) dy$$
Where $f(Y|X)$ is the conditional density. Using the product rule, you can show:
$$f(y|x)=\frac{f(y,x)}{f(x)}$$
Substituting this back into the integral you get
$$E(Y|X)\equiv\frac{\int y f(y,x) dy}{f(x)}$$
Which is of the form you seek, if you use the kernel density estimator.
3
You mean for a random variable with a single Epanechnikov kernel as PDF? Well, the PDF is $\frac{3}{4}(1-u^2)$, so the CDF is $\frac{1}{4}(2 + 3 u - u^3)$. Inverting this in Maple leads to three solutions, of which $u = -1/2\,{\frac { \left( 1-2\,t+2\,i\sqrt {t}\sqrt {1-t} \right) ^{2/3}+1 +i\sqrt {3} \left( 1-2\,t+2\,i\sqrt {t}\sqrt {1-t} \right) ^{2/3}-i
...
3
A quick google brings up this, which indicates that when working with circular data you'll need a different definition of 'bias' for a start:
However, when using data on the circle, we cannot use distance in Euclidean space, so all differences θ − θi should be replaced by considering the angle between two vectors:
$d_i\theta)= \| \theta -\theta_i \| ...
Only top voted, non community-wiki answers of a minimum length are eligible
