Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am to use the Squared Exponential kernel (SE) for Gaussian Process Regression. The advantages of this kernel are: 1) simple: only 3 hyperparameters; 2) smooth: this kernel is Gaussian.

Why do people like 'smoothness' so much? I know that the Gaussian kernel is infinitely differentiable, but is that so important? (Please let me know if there is other reasons why the SE kernel is so popular.)

PS: I was told that most signals in real world (without noise) are smooth, so it is reasonable to use smooth kernels to model them. Could anyone please help me understand this concept?

share|improve this question
4  
Are you asking the psychological question of why people like smoothness or the statistical question of why smooth functions are better statistically? –  John Aug 4 at 12:00
    
@John Thank you for your comment. I was refering to the second question in your post and in addition, I want to confirm why are most signal in real world are smooth –  kakanana Aug 4 at 20:22

3 Answers 3

"Natura non facit saltus" is an old principle in philosophy. Also, beauty and harmony are such principles. Another philosophical principle that has impact on statistics is qualitative thinking: Traditionally we don't think in effect sizes but whether an effect is there or not. This let to hypothesis testing. Estimators are too precise for your perception of nature. Take it as it is.

Statistics has to serve the human perception. So discontinuity points are disliked. One would immediately ask: Why is exactly at this a discontinuity? Especially in density estimation, these discontinuity points are mostly due to non asymptotical nature of real data. But you don't want to learn about your certain finite sample but about the underlying natural fact. If you believe this nature doesn't jump, then you need smooth estimators.

From a strict mathematical point of view, there is hardly a reason for it. Also, since Leibniz and Newton natural phenomena became known that are not smooth. Talk to the natural scientist you're working for. Challenge his view of smoothness/discontinuity and then do what you both decided to be most helpful for his understanding.

share|improve this answer

There are many motivations, depending on the problem. But the idea is the same: add a priori knowledge about some problem to achieve a better solution and cope with complexity. A more way to put it is: model selection. Here a nice example on model selection.

Another idea, deeply related to it is to find a similarity measure of data samples (there are different terms that relate to that idea: topographical mappings, distance metric, manifold learning,...).

Now, let us consider a practical example: optical character recognition. If you take the image of a character, you would expect the classifier to deal with invariances: if you rotate, displace or scale the image, it should be able to detect it. Also, if you apply some one modification slightly to the input, you would expect the answer/behaviour of your classifier to vary just slightly as well, because both samples (the original and the modified are very similar). This is where the enforcement of smoothness comes in.

There is a wealth of papers dealing with this idea, but this one (transformation invariance in pattern recognition, tangent distance and tangent propagation, Simard et. al) illustrates these ideas in great detail

share|improve this answer

There are two more reasons of practical matters. The first one is that analytical functions are much easier to work with mathematically, and therefore prove theorems about your algorithms and give them a stronger foundation.

The second is sensitivity. Say you have a machine learner $M$ whose output has a discontinuity at $x=x_0$. Then you would get very different results for $x_0 - \epsilon$ and $x_0 + \epsilon$, but that's OK because we made it discontinuous. Now, if you train your model with slightly different data ($\tilde M$), where the random noise is just a tiny bit different, the discontinuity will now be at $\tilde x_0$, probably very close to $x_0$, but not quite, and now, for some values of $\epsilon$, $x_0 + \epsilon$ has a very different value for $M$ and for $\tilde M$.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.