I am reading Chapter 11 of Elements of Statistical learning and came across this sentence:
"Unlike methods like CART and MARS, neural networks are smooth functions of real-valued parameters"
What is meant by 'smooth functions' here? I have come across things such as smoothing splines, but am unsure what a 'smooth function' means more generally.
Following on from the above, what makes neural networks specifically smooth functions?
A smooth function has continuous derivatives, up to some specified order. At the very least, this implies that the function is continuously differentiable (i.e. the first derivative exists everywhere and is continuous). More specifically, a function is $C^k$ smooth if the 1st through $k$th order derivatives exist everywhere, and are continuous.
Neural nets can be written as compositions of elementary functions (typically affine transformations and nonlinear activation functions, but there are other possibilities). For example, in feedforward networks, each layer implements a function whose output is passed as input to the next layer. Historically, neural nets have tended to be smooth, because the elementary functions used to construct them were themselves smooth. In particular, nonlinear activation functions were typically chosen to be smooth sigmoidal functions like $\tanh$ or the logistic sigmoid function.
However, the quote is not generally true. Modern neural nets often use piecewise linear activation functions like the rectified linear (ReLU) activation function and its variants. Although this function is continuous, it's not smooth because the derivative doesn't exist at zero. Therefore, neural nets using these activation functions are not smooth either.
In fact, the quote isn't generally true, even historically. The McCulloch-Pitts model was the first artificial neural net. It was composed of thresholded linear units, which output binary values. This is equivalent to using a step function as the activation function. This function isn't even continuous, let alone smooth.
They refer to the smoothness, as understood in mathematics, so a function that is continuous and differentiable. As explained by Nick S on math.stackexchange.com:
A function being smooth is actually a stronger case than a function being continuous. For a function to be continuous, the epsilon delta definition of continuity simply needs to hold, so there are no breaks or holes in the function (in the 2-d case). For a function to be smooth, it has to have continuous derivatives up to a certain order, say k.
Some of the answers at math.stackexchange.com mention infinite differentiability, but in machine learning the term would be rather used in looser sense of not-necessary-infinite differentiability, since we rather wouldn't need infinite differentiability for anything.
This can be illustrated using the figure used on scikit-learn site (below), showing decision boundaries of different classifiers. If you look at decision tree, random forest, or AdaBoost, the decision boundaries are overlayed rectangles, with sharp, rapidly changing boundaries. For neural network, the boundary is smooth both in mathematical sense and in common, everyday sense, where we say that something is smooth, i.e. something rather roundish, without sharp edges. Those are decision boundaries of classifiers, but regression analogs of those algorithms work almost the same.
Decision tree is an algorithm that outputs a number of, automatically generated, if ... else ... statements that lead to final nodes where it makes the final prediction, e.g. if age > 25 and gender = male and nationality = German then height = 172 cm. By design, this would produce predictions that are characterized by "jumps", because one node would predict height = 172 cm while other height = 167 cm and there might be nothing in-between.
MARS regression is build in terms of piecewise linear units with "breaks", so the regression equation when using single feature $x$, and two breaks, could be something like below
$$ y = b + w_1 \max(0, x - a_1) + w_2 \max(0, x - a_2) $$
notice that the $\max$ function is an element that is continuous, but not differentiable (it is even used as an example in Wikipedia), so the output would not be smooth.
Neural networks are build in terms of layers, where each layer is build from neurons like
$$ h(x) = \sigma(wx + b) $$
so when the neurons are smooth, the output would be smooth as well. Notice however that if you used neural network with one hidden layer using two neurons, $\operatorname{ReLU}(x) = \max(0, x)$ activation on hidden layer, and linear activation on output layer, then the network could be something like
$$ \newcommand{\relu}{\operatorname{ReLU}} y = b + w^{(2)}_1 \relu(w^{(1)}_1 x + a_1) + w^{(2)}_2 \relu(w^{(1)}_2 x + a_2) $$
that is almost the same model as MARS, so isn't smooth as well... There are also other examples where modern neural networks architectures do not need to lead to smooth solutions, so the statement is not generally true.
When the book was written nobody was using relu . It’s not even mentioned in the book. All activations were smooth sigmoids. In this case neural net output is indeed a smooth function of its parameters such as weights and biases. That’s how you make backpropagation work nicely but slowly. Once relu came to the picture derivatives calculations became much faster, because it became piecewise linear instead of smooth nonlinear
