Are "covariance function" and "kernel function" synonyms? Are "covariance function" and "kernel function" synonyms? In the Gaussian process (GP) literature, authors typically discuss covariance functions, whereas in the support vector machine (SVM) literature, authors typically discuss kernel functions. It appears that both refer to the same underlying objects, as the resulting covariance matrix is a Mercer kernel: both are symmetric, positive semidefinite matrices. Is the difference in terminology purely due to the historical development of the respective tools, or is there a subtle shade of meaning that I'm missing?
 A: Yes, "covariance function" and "positive-definite kernel" refer to the same concept. (Authors in the SVM literature sometimes omit the qualification "positive-definite", since it's typically by far the most relevant type of kernel.)
For example, see page 80 of Rasmussen and Williams, Gaussian Processes for Machine Learning, 2006.
A: Danica's answer is correct. Precisely stated, covariance matrices and Mercer kernels are both matrices which are (1) positive definite and (2) symmetric. However, there is some research into matrices otherwise than Mercer kernels, that is, matrices which are not positive-definite but which may be useful in machine learning nonetheless. These are occasionally referred to as kernels, as in this paper, but at least in this case the authors are careful to stress that when they are speaking of kernels, they have a definition in mind which does not necessarily satisfy Mercer's conditions.
Cheng Soon, Xavier Mary, Alexander J. Smola. "Learning with Non-Positive Kernels." Appearing in Proceedings of the 21 st International Conference
on Machine Learning, Banff, Canada, 2004.
