8
$\begingroup$

I am currently studying the textbook Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams. Chapter 1 Introduction says the following:

Given this training data we wish to make predictions for new inputs $\mathbf{\mathrm{x}_*}$ that we have not seen in the training set. Thus it is clear that the problem at hand is inductive; we need to move from the finite training data $\mathcal{D}$ to a function $f$ that makes predictions for all possible input values. To do this we must make assumptions about the characteristics of the underlying function, as otherwise any function which is consistent with the training data would be equally valid. A wide variety of methods have been proposed to deal with the supervised learning problem; here we describe two common approaches. The first is to restrict the class of functions that we consider, for example by only considering linear functions of the input. The second approach is (speaking rather loosely) to give a prior probability to every possible function, where higher probabilities are given to functions that we consider to be more likely, for example because they are smoother than other functions.

I am curious about this part:

The second approach is (speaking rather loosely) to give a prior probability to every possible function, where higher probabilities are given to functions that we consider to be more likely, for example because they are smoother than other functions.

Why does a function being smoother make it more likely?

$\endgroup$
5
  • $\begingroup$ I think you are missing the point: they are just distinguishing between a hard constraint and a soft constraint: favouring certain classes of functions over others. The 'likeliness' is just a mathematical way of expressing this soft constraint ( not the other way around) $\endgroup$ – seanv507 Dec 25 '20 at 23:18
  • 2
    $\begingroup$ "Why does a function being smoother make it more likely?" This is not generally true. It is only true for specific contexts. Are you asking for specific contexts where this would be true? I believe that makes this question too broad and also off-topic (not about statistics). $\endgroup$ – Sextus Empiricus Dec 26 '20 at 17:46
  • $\begingroup$ @SextusEmpiricus I am asking within the context of the quoted textbook. $\endgroup$ – The Pointer Dec 27 '20 at 1:32
  • $\begingroup$ @ThePointer in the textbook it is given as an example for a restriction on functions (without giving context why this would need to be a restriction). It is not a general property of functions that we consider them more likely if they are smoother. $\endgroup$ – Sextus Empiricus Dec 27 '20 at 8:25
  • $\begingroup$ I think the discussion and some answers below miss the most important point: Your concrete, specific and entirely non-philosophical knowledge of your domain of application. If you are a Geostatistician and would like to model this you would have a clear opinion whether a very smooth or a more rugged model of differentiability is probable. $\endgroup$ – g g Jan 6 at 15:59
8
$\begingroup$

While the author mentions it as an "example", it is true that, generally, smoother functions are often preferred in modelling the characteristics of the "true" underlying function, and therefore may be assigned a higher "prior probability", as the author maintains. Why is this? You may learn more about it by reading this similar question here, but essentially, there is no real justification for it, just the conventional belief that most things occurring in nature tend to change gradually rather than in a non-continuous way. Practically, smoother functions are desired because they are more easily differentiated and may have convenient mathematical properties. More on that discussion here.

However, though I would say that smooth functions are still widely anchored in statistical methods, in my experience over the years we have been working more and more with non-smooth functions. Examples I can think of include in the context of real-world optimization problems, interpolation problems, and many applications of deep neural networks (an easy one to see is the common ReLU activation function).

In any case, while this question easily spurs debate, I think opportunities to ponder underlying principles are great!

$\endgroup$
2
  • 3
    $\begingroup$ I've seen people argue that smooth functions are preferable because small changes in the inputs result in small changes in the outputs. This is in the spirit of your argument that "things occurring in nature tend to change gradually rather than in a non-continuous way". +1 $\endgroup$ – Demetri Pananos Dec 25 '20 at 18:26
  • 3
    $\begingroup$ @DemetriPananos But this is a property of continuous functions, not of smooth functions (although sometimes differentiability can helpt to quantify this). $\endgroup$ – Jannik Pitt Dec 25 '20 at 18:54
8
+50
$\begingroup$

One intuitive way to view it is that a smooth function can be described with less information than a less smooth function. If we restrict ourselves to vector spaces of functions, the dimension of the vector space (finite or infinite) is the number of coefficients we need to give for a complete specification of the function. For a linear function we need two coefficients, the slope and intercept. So for random linear function, we must specify a 2-dim joint distribution on the slope and intercept. For more wiggly functions we need higher-dimensional joint distributions, so intuitively the probability mass is more "spread out" over a larger volume in parameter space (some would say state space), so total probability is spread over "more functions", and then probability densities must be lower. That implies, in particular, if those functions serve as parameter in some likelihood function, likelihood will be more spread out, the densities will be lower and vary slower, so in particular, Fisher information will be lower.

Let us see how this works out for some simulated spline functions$^\dagger$. First I show a plot of the spline basis functions for a natural spline with 5 degrees of freedom:

spline basis functions

Then we can simulate some actual spline functions by drawing standard normal coefficients randomly:

Some simulated spline functions

If we had chosen, say, more interior knots, we would have got wigglier, less smooth, functions, which need more information to be described.

$^\dagger$Splines are piecewise polynomials, knots are the points where we shift from one poly to the next. We could have used other functions as an example, even just polynomials. See
Splines - basis functions - clarification Interpretation of a spline and search this site.

For reference, the actual R code used:

library(splines)

x <- 1:20
S <- ns(x, knots=c(5, 10,  15),
        intercept=TRUE, Boundary.knots=c(0, 21))

# Plot the basis functions:

library(ggplot2)
library(reshape2)
 pframe <- melt(as.data.frame(S), measure.vars=1:5)  
 pframe$x <- rep(x, 5)

ggplot( pframe, aes(x=x, y=value, group=variable, color=variable)) +
    geom_line() + ggtitle("Natural spline basis functions")

# Then we can simulate some coefficients and plot the resulting functions: 
# First we choose the coefficients as iid standard normal:

set.seed(7*11*13)# My public seed

N <- 5
n <- length(x)
simfuns <- data.frame(x=rep(x, N), Y=as.numeric(NA), group=rep(1:N, each=n))

for (i in 1:N) simfuns$Y[((i-1)*n+1):(i*n)] <- S %*% rnorm(5)

ggplot(simfuns, aes(x=x, y=Y, group=group, color=group)) +
    geom_line() + ggtitle("Some simulated spline functions:")

$\endgroup$
11
  • $\begingroup$ Interesting answer. What do you mean by "interior knots", and why would this produce "wigglier, less smooth, functions"? $\endgroup$ – The Pointer Dec 27 '20 at 8:34
  • 2
    $\begingroup$ Yes, the knots are the corners where (many) derivatives are undefined. Mostly used (also my example) is cubic splines, where the knots have first & second derivatives, but not more $\endgroup$ – kjetil b halvorsen Dec 27 '20 at 23:35
  • 2
    $\begingroup$ I think this is a great answer, +6. $\endgroup$ – gung - Reinstate Monica Dec 30 '20 at 16:45
  • 2
    $\begingroup$ There's also a related point that if we find ourselves with practical restrictions like a finite (i.e. in practice: always too small) data set and the requirement to get reasonably certain estimates of the coefficients for practical purposes, we cannot afford the more wiggly functions. This doesn't imply a strict need to restrict ourselves, though: the width of the credible intervals would tell us that we tried to estimate something we couldn't really afford. $\endgroup$ – cbeleites unhappy with SX Dec 30 '20 at 20:54
  • 1
    $\begingroup$ @gg: I'm refering to the uncertainty in the estimated function coefficients, not (or only indirectly) about the posterior probability distribution. (Or do I also misunderstand you?) Let me reformulate my point: this may be practical heuristic: you can calculate via data that you cannot afford too wiggly functions. You can either exlude them entirely from consideration, or take a somewhat softer approach and incorporate the external knowledge that we're less likely to be able to afford them into the prior. $\endgroup$ – cbeleites unhappy with SX Jan 7 at 18:58
4
$\begingroup$

I disagree with the other answers here asserting that there is no good reason for this, and that it is merely a simplifying assumption. From a metaphysical perspective, causal effects in nature generally operate in a roughly "smooth" manner, and so small changes in the input quantities in a causal system generally result in small changes in the output. Of course, this is not always the case; there are some causal systems that exhibit large changes with threshold effects, and there are some chaotic systems where small chnages in inputs may lead to large and unpredictable changes in outputs. However, as a general rule, causal changes exhibit smoothness between inputs and outputs. This is a metaphysical property of nature, and not merely a modelling or statistical convention. One can certainly note that this is not true in all cases, but it is true in most applications where we model related variables.

For example, when you throw a ball in the air (without any obstruction above you), the force you impart to the ball affects the height it reaches in a smooth manner. If you throw it slightly harder it will go slightly higher, and so forth. Similarly, if a gust of upward or downward wind affects the trajectory of the ball, the wind-speed and angle will affect the height the ball reaches in a roughly smooth manner. If you have a slightly stronger wind it will affect the ball height slightly more, and so forth. I have given physical examples for simplicity, but similar outcomes occur in a range of areas including economics, finance, psychology, etc.

There is an "anthropomorphic" philosophical argument that can be made here. If the universe were such that causal laws tended not to be "smooth" then it would be a very chaotic place, and it is unlikely that life could exist; a fortiori intelligent life. Hence, our presence as living cognisant observers asking this question constitutes a form of selection bias that virtually necessitates smooth "well behaved" causal laws.

There is also a complication here in what we even regard to be "the function" we are estimating in the first place. Real-life problems involve a finite set of outcomes in nature, even in large populations, so the use of a mathematical function over a continuum is already an abstraction that goes beyond the observable data. We generally posit that natural forces can exist on a continuum and that physical/natural laws can likewise be properly described by continuous functions (e.g., this is the methodology in physics), but this can become more tenuous when we are looking at phenomena that are specific rather than general. Thus, your question is pregnant with some deeper metaphysical and epistemological quesstions about the validity of approximating finite sets of outcomes in nature by infinite/continuous mathematical descriptions.$^\dagger$

As you can see, a seemingly simple question like this opens up a lot of interesting philosophical doors. If you would like to learn more about these issues, I recommend reading some material on finitism by Doron Zeilberger and some material on the anthropic principle by Nick Bostrom.


$^\dagger$ Indeed, you should not take for granted the use of continuous functions in mathematics at all. There are a number of philosophers/mathematicians who object to the use of infinite mathematical objects (see e.g., finitism, ultrafinitism). To these practitioners, the very notion that there is a function on a "continuum" is already flawed.

$\endgroup$
8
  • 2
    $\begingroup$ I don't find any disagreement of this with my answer, they are complementary, not contradictory ... (+1) (when I have new votes ... ) $\endgroup$ – kjetil b halvorsen Dec 27 '20 at 23:16
  • 1
    $\begingroup$ Interesting contribution. It seems to me that Sharon Choong's answer agrees with your first paragraph, no? $\endgroup$ – The Pointer Dec 27 '20 at 23:32
  • 1
    $\begingroup$ No --- according to her answer, "essentially, there is no real justification for it". $\endgroup$ – Ben Dec 28 '20 at 0:20
  • $\begingroup$ When one uses a squared exponential kernel for a Gaussian process then one assumes that the functions must be infinitely differentiable. That is a very restricted assumption. I don't really see this anthropic principle (or the argument "so small changes in the input quantities in a causal system generally result in small changes in the output") as a good argument for that. Maybe continuous or a few times differentiable functions can be nice to describe laws of nature, but that's an argument for continuous or differentiable functions and not an argument that more smoother is more likely. $\endgroup$ – Sextus Empiricus Dec 28 '20 at 21:36
  • $\begingroup$ If it is an argument for continuity or differentiability (as opposed to non-continuity/non-differentiability) then it is already an argument for greater smoothness. $\endgroup$ – Ben Dec 28 '20 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.