Why does a function being smoother make it more likely?

Question

I am currently studying the textbook Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams. Chapter 1 Introduction says the following:

Given this training data we wish to make predictions for new inputs $\mathbf{\mathrm{x}_*}$ that we have not seen in the training set. Thus it is clear that the problem at hand is inductive; we need to move from the finite training data $\mathcal{D}$ to a function $f$ that makes predictions for all possible input values. To do this we must make assumptions about the characteristics of the underlying function, as otherwise any function which is consistent with the training data would be equally valid. A wide variety of methods have been proposed to deal with the supervised learning problem; here we describe two common approaches. The first is to restrict the class of functions that we consider, for example by only considering linear functions of the input. The second approach is (speaking rather loosely) to give a prior probability to every possible function, where higher probabilities are given to functions that we consider to be more likely, for example because they are smoother than other functions.

I am curious about this part:

The second approach is (speaking rather loosely) to give a prior probability to every possible function, where higher probabilities are given to functions that we consider to be more likely, for example because they are smoother than other functions.

Why does a function being smoother make it more likely?

Answer 1

While the author mentions it as an "example", it is true that, generally, smoother functions are often preferred in modelling the characteristics of the "true" underlying function, and therefore may be assigned a higher "prior probability", as the author maintains. Why is this? You may learn more about it by reading this similar question here, but essentially, there is no real justification for it, just the conventional belief that most things occurring in nature tend to change gradually rather than in a non-continuous way. Practically, smoother functions are desired because they are more easily differentiated and may have convenient mathematical properties. More on that discussion here.

However, though I would say that smooth functions are still widely anchored in statistical methods, in my experience over the years we have been working more and more with non-smooth functions. Examples I can think of include in the context of real-world optimization problems, interpolation problems, and many applications of deep neural networks (an easy one to see is the common ReLU activation function).

In any case, while this question easily spurs debate, I think opportunities to ponder underlying principles are great!

Answer 2

One intuitive way to view it is that a smooth function can be described with less information than a less smooth function. If we restrict ourselves to vector spaces of functions, the dimension of the vector space (finite or infinite) is the number of coefficients we need to give for a complete specification of the function. For a linear function we need two coefficients, the slope and intercept. So for random linear function, we must specify a 2-dim joint distribution on the slope and intercept. For more wiggly functions we need higher-dimensional joint distributions, so intuitively the probability mass is more "spread out" over a larger volume in parameter space (some would say state space), so total probability is spread over "more functions", and then probability densities must be lower. That implies, in particular, if those functions serve as parameter in some likelihood function, likelihood will be more spread out, the densities will be lower and vary slower, so in particular, Fisher information will be lower.

Let us see how this works out for some simulated spline functions$^\dagger$. First I show a plot of the spline basis functions for a natural spline with 5 degrees of freedom:

Then we can simulate some actual spline functions by drawing standard normal coefficients randomly:

If we had chosen, say, more interior knots, we would have got wigglier, less smooth, functions, which need more information to be described.

$^\dagger$Splines are piecewise polynomials, knots are the points where we shift from one poly to the next. We could have used other functions as an example, even just polynomials. See
Splines - basis functions - clarification Interpretation of a spline and search this site.

For reference, the actual R code used:

library(splines)

x <- 1:20
S <- ns(x, knots=c(5, 10,  15),
        intercept=TRUE, Boundary.knots=c(0, 21))

# Plot the basis functions:

library(ggplot2)
library(reshape2)
 pframe <- melt(as.data.frame(S), measure.vars=1:5)  
 pframe$x <- rep(x, 5)

ggplot( pframe, aes(x=x, y=value, group=variable, color=variable)) +
    geom_line() + ggtitle("Natural spline basis functions")

# Then we can simulate some coefficients and plot the resulting functions: 
# First we choose the coefficients as iid standard normal:

set.seed(7*11*13)# My public seed

N <- 5
n <- length(x)
simfuns <- data.frame(x=rep(x, N), Y=as.numeric(NA), group=rep(1:N, each=n))

for (i in 1:N) simfuns$Y[((i-1)*n+1):(i*n)] <- S %*% rnorm(5)

ggplot(simfuns, aes(x=x, y=Y, group=group, color=group)) +
    geom_line() + ggtitle("Some simulated spline functions:")

Answer 3

I disagree with the other answers here asserting that there is no good reason for this, and that it is merely a simplifying assumption. From a metaphysical perspective, causal effects in nature generally operate in a roughly "smooth" manner, and so small changes in the input quantities in a causal system generally result in small changes in the output. Of course, this is not always the case; there are some causal systems that exhibit large changes with threshold effects, and there are some chaotic systems where small chnages in inputs may lead to large and unpredictable changes in outputs. However, as a general rule, causal changes exhibit smoothness between inputs and outputs. This is a metaphysical property of nature, and not merely a modelling or statistical convention. One can certainly note that this is not true in all cases, but it is true in most applications where we model related variables.

For example, when you throw a ball in the air (without any obstruction above you), the force you impart to the ball affects the height it reaches in a smooth manner. If you throw it slightly harder it will go slightly higher, and so forth. Similarly, if a gust of upward or downward wind affects the trajectory of the ball, the wind-speed and angle will affect the height the ball reaches in a roughly smooth manner. If you have a slightly stronger wind it will affect the ball height slightly more, and so forth. I have given physical examples for simplicity, but similar outcomes occur in a range of areas including economics, finance, psychology, etc.

There is an "anthropomorphic" philosophical argument that can be made here. If the universe were such that causal laws tended not to be "smooth" then it would be a very chaotic place, and it is unlikely that life could exist; a fortiori intelligent life. Hence, our presence as living cognisant observers asking this question constitutes a form of selection bias that virtually necessitates smooth "well behaved" causal laws.

There is also a complication here in what we even regard to be "the function" we are estimating in the first place. Real-life problems involve a finite set of outcomes in nature, even in large populations, so the use of a mathematical function over a continuum is already an abstraction that goes beyond the observable data. We generally posit that natural forces can exist on a continuum and that physical/natural laws can likewise be properly described by continuous functions (e.g., this is the methodology in physics), but this can become more tenuous when we are looking at phenomena that are specific rather than general. Thus, your question is pregnant with some deeper metaphysical and epistemological quesstions about the validity of approximating finite sets of outcomes in nature by infinite/continuous mathematical descriptions.$^\dagger$

As you can see, a seemingly simple question like this opens up a lot of interesting philosophical doors. If you would like to learn more about these issues, I recommend reading some material on finitism by Doron Zeilberger and some material on the anthropic principle by Nick Bostrom.

---

$^\dagger$ Indeed, you should not take for granted the use of continuous functions in mathematics at all. There are a number of philosophers/mathematicians who object to the use of infinite mathematical objects (see e.g., finitism, ultrafinitism). To these practitioners, the very notion that there is a function on a "continuum" is already flawed.