# Tag Info

13

The optimal bandwidth for derivative estimation will be different from the bandwidth for density estimation. In general, every feature of a density has its own optimal bandwidth selector. If your objective is to minimize mean integrated squared error (which is the usual criterion) there is nothing subjective about it. It is a matter of deriving the value ...

12

Zen used method 1. Here is method 2: Map $x$ to a spherically symmetric Gaussian distribution centered at $x$ in the Hilbert space $L^2$. The standard deviation and a constant factor have to be tweaked for this to work exactly. For example, in one dimension, $$\int_{-\infty}^\infty \frac{\exp[-(x-z)^2/(2\sigma^2)]}{\sqrt{2 \pi} \sigma} ... 12 One solution, borrowed from approaches to edge-weighting of spatial statistics, is to truncate the density on the left at zero but to up-weight the data that are closest to zero. The idea is that each value x is "spread" into a kernel of unit total area centered at x; any part of the kernel that would spill over into negative territory is removed and ... 11 For a univariate KDE, you are better off using something other than Silverman's rule which is based on a normal approximation. One excellent approach is the Sheather-Jones method, easily implemented in R; for example, plot(density(precip, bw="SJ")) The situation for multivariate KDE is not so well studied, and the tools are not so mature. Rather than a ... 9 It depends a little bit on what your end goal is. Quick and dirty hack for real-time demonstrations Using Sys.sleep(seconds) in a loop where seconds indicates the number of seconds between frames is a viable option. You'll need to set the xlim and ylim parameters in your call to plot to make things behave as expected. Here's some simple demonstration ... 9 I think the main problem is to get the pairwise distances efficiently. Once you have that the rest is element wise. To do this, you probably want to use scipy. The function scipy.spatial.distance.pdist does what you need, and scipy.spatial.distance.squareform will possibly ease your life. So if you want the kernel matrix you do from scipy.spatial.distance ... 8 To shamelessly quote the Stata manual entry for kdensity: The optimal width is the width that would minimize the mean integrated squared error if the data were Gaussian and a Gaussian kernel were used, so it is not optimal in any global sense. In fact, for multimodal and highly skewed densities, this width is usually too wide and oversmooths the density ... 8 For x,y on S, certain functions K(x,y) can be expressed as an inner product (in usually a different space). K is often referred to as a kernel or a kernel function. The word kernel is used in different ways throughout mathematics, but this is the most common usage in machine learning. The kernel trick is a way of mapping observations from a general set S ... 8 I will use method 1. Check Douglas Zare's answer for a proof using method 2. I will prove the case when x,y are real numbers, so k(x,y)=\exp(-(x-y)^2/2\sigma^2). The general case follows mutatis mutandis from the same argument, and is worth doing. Without loss of generality, suppose that \sigma^2=1. Write k(x,y)=h(x-y), where ... 8 The Moore-Aronszajn theorem guarantees that a symmetric positive definite kernel is associated to a unique reproducing kernel Hilbert space. (Note that while the RKHS is unique, the mapping itself is not.) Therefore, your question can be answered by exhibiting an infinite-dimensional RKHS corresponding to the Gaussian kernel (or RBF). You can find an ... 7 An alternative is the approach of Kooperberg and colleagues, based on estimating the density using splines to approximate the log-density of the data. I'll show an example using the data from @whuber's answer, which will allow for a comparison of approaches. set.seed(17) x <- rexp(1000) You'll need the logspline package installed for this; install it ... 6 Do your analysis with several different kernels. Make sure you cross-validate. Choose the kernel that performs the best during cross-validation and fit it to your whole dataset. /edit: Here is some example code in R, for a classification SVM: #Use a support vector machine to predict iris species library(caret) library(caTools) #Choose x and y x <- ... 6 Basically anything what is not separable with a line (ok, hyperplane), for instance 2D data like this: kernel trick will effectively project this situation into a (higher-dim) space in which linear separation is possible; see this movie for an effect of a gaussian kernel on similar data. Look for a kernel argument in your svm function ;-) Note that using a ... 6 One way to go is to use the excellent animation package by Yihui Xie. I uploaded a very simple example to my public dropbox account: densityplot (I will remove this example in 3 days). Is this what you are looking for? The animation was created using the following R code: library(animation) density.ani <- function(){ i <- 1 d <- ... 6 You are right about the three issues you raise, and your interpretation is exactly right. People have looked at other directions to build kernels from probabilistic models: Moreno et al. propose Kullback-Leibler although when this satisfies Mercer's conditions was not well understood when I looked at this problem back when I read it. Jebara et al. ... 6 I've written a couple ;o) G. C. Cawley and N. L. C. Talbot, Efficient approximate leave-one-out cross-validation for kernel logistic regression, Machine Learning, vol, 71, no. 2-3, pp. 243--264, June 2008. Which gives a reasonable method for choosing kernel and regularisation parameters and an empirical evaluation G. C. Cawley, G. J. Janacek and N. L. C. ... 6 The answering question is "why do you model your data as a sample from a distribution?" If you want to learn something about the phenomenon behind your data, like when improving a scientific theory or testing a scientific hypothesis, using a non-parametric kernel estimator does not tell you much more than the data istself. While a parameterised model can ... 6 I don't use R routinely and I have never used ggplot, but there is a simple story here, or so I guess. Time of day is manifestly a circular or periodic variable. In your data you have hours 0(1)23 which wrap around, so that 23 is followed by 0. However, ggplot does not know that, at least from the information you have given it. So far as it is concerned ... 5 In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y. 5 No, this kernel doesn't give a positive (semi-) definite Gram matrix, so it is not a valid kernel (the first randomly generated Gram matrix \matrix{K} and vector \vec{v} I tried gave \vec{v}'\matrix{K}\vec{v} < 0 so \matrix{K} isn't positive semi-definite. If you infer from that that I am an engineer - you are correct! ;o). ISTR that Kernel ... 5 Grid search is a sensible procedure as @JohnSmith suggests, however it is not the only stable technique. I generally use the Nelder-Mead simplex algortihm, which I have found to be very reliable and more efficient than grid search as less time is spent investigating areas of hyper-parameter space that give poor models. If you are a MATLAB user, you can get ... 5 If you look at the optimization problem that SVM solves: \min_{\mathbf{w},\mathbf{\xi}, b } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^n \xi_i \right\} s.t. y_i(\mathbf{w}\cdot\mathbf{x_i} - b) \ge 1 - \xi_i, ~~~~\xi_i \ge 0, for all  i=1,\dots n the support vectors are those x_i where the corresponding \xi_i \gt 0. In other words, ... 5 In a machine learning context (i.e. "kernel methods"), the key requirement for a kernel is that it must be symmetric and positive-definite, that is, if K is a kernel matrix, then for any (column) vector x of the appropriate length, x^{T}Kx must be a positive real number. This restriction is in place mostly due to requirements of optimization processes ... 5 You can obtain the explicit equation of \phi for the Gaussian kernel via the Tailor series expansion of e^x. For notational simplicity, assume x\in \mathbb{R}^1:$$\phi(x) = e^{-x^2/2\sigma^2} \Big[ 1, \sqrt{\frac{1}{1!\sigma^2}}x,\sqrt{\frac{1}{2!\sigma^4}}x^2,\sqrt{\frac{1}{3!\sigma^6}}x^3,\ldots\Big]^T$$This is also discussed in more detail in ... 5 Usually, the decision is whether to use linear or an RBF (aka Gaussian) kernel. There are two main factors to consider: Solving the optimisation problem for a linear kernel is much faster, see e.g. LIBLINEAR. Typically, the best possible predictive performance is better for a nonlinear kernel (or at least as good as the linear one). It's been shown that ... 5 Corresponding to any batch of data X = (x_1, x_2, \ldots, x_n) is its "empirical density function"$$f_X(x) = \frac{1}{n}\sum_{i=1}^{n} \delta(x-x_i).$$Here, \delta is a "generalized function;" its defining property is that for any continuous function g of compact support,$$\int_{\mathbb{R}}\delta(x) g(x) dx = g(0). (Names for $\delta$ include ...

4

There appear to be at least two different meanings of "kernel": one more commonly used in statistics; the other in machine learning. In statistics "kernel" is most commonly used to refer to kernel density estimation and kernel smoothing. A straightforward explanation of kernels in density estimation can be found (here). In machine learning "kernel" is ...

4

Two further references from B. Schölkopf: Schölkopf, B. and Smola, A.J. (2002). Learning with kernels. The MIT Press. Schölkopf, B., Tsuda, K., and Vert, J.-P. (2004). Kernel methods in computational biology. The MIT Press. and a website dedicated to kernel machines.

4

You can use conditional kernel density estimation to obtain the density of sales at time $t+h$ conditional on the values of sales at times $t, t-1, t-2, \dots$ This gives you a density forecast rather than a point forecast. The problem is that the conditioning is difficult in a density setting when the number of conditioning variables is more than 2. See ...

4

I asked a similar question a few months ago. Rob Hyndman provided an excellent answer that recommends the Sheather-Jones method. One addition point. In R, for the density function, you set the bandwidth explicitly via the bw argument. However, I often find that the adjust argument is more helpful. The adjust argument scales the value of the bandwidth. So ...

Only top voted, non community-wiki answers of a minimum length are eligible