How to use "kernel trick" in Stochastic gradient descent? How to use "kernel trick" in Stochastic gradient descent? I can find kernel perceptron on Wikipedia but I can't find "kernel sgd" anywhere that gives me a clear algorithm to do that. Can someone teach me how I can apply kernel in SGD? 
 A: AFAIK the kernel trick has pretty much nothing to do with SGD. Are you thinking SVM instead of SGD perhaps? The kernel trick is an optimization used mainly to compute SVM models.
The "Kernel Trick" is a method that allows a high dimensional feature space to be fit using what is called an inner product space instead of explicitly computing that high-dimensional features space. This allows SVMs to be fit using nonlinear features such as radial basis functions and polynomials. In practice, this provides nonlinear capabilities to your model while still enjoying SVM's global convergence.
Nonlinear features can also be fit using gradient descent, which provides much greater flexibility in nonlinear features. However, it can only gaurantee local convergence, and often even local convergence is time consuming. To improve the search, a method called stochastic gradient descent is sometimes used, especially in larger datasets. The basic idea is, for example, if you have a million examples, you can train more effectively with gradient descent batches of randomly chosen subsets of size 1000 than you can with the entire million in each iteration. 1000 still describes much of the range of the data, and a million is computationally inefficient to run through each iteration.
One possible link between these concepts could also be L-BFGS training. It is a higher-order training method that can be used in place of stochastic gradient descent, but (somewhat) similar to the kernel trick it uses a method to approximate the Hessian using a memory of gradient vectors. This is a bit of a stretch, but I thought worth mentioning.
A: Please have a look at  this paper to know how to do SGD in RKHS, which is same as what you are asking for.
