In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning:
$$\text{RSS}(f,\lambda) = \sum_{i=1}^{N} \{y_i - f(x_i)\}^2 + \lambda \int \{f''(t)\}^2 dt$$
I've included a screenshot of the relevant paragraph here:
Welcome to CV!
As the authors mention, $f$ can be any interpolating function. The goal of the regularizing term is to penalize the function for tortuosity, as excessive twisting and curving to match every peculiarity in the data is likely to result in overfitting, picking up peculiarities of the sample that are unlikely to be present in its population. Namely, the function that minimizes the unpenalized loss:
$$\sum_{i=1}^N\big( y_i - f(x_i) \big)^2,$$
does exactly that for a sufficiently complex $f$.
Hence, the loss function is penalized by the regularizing term:
$$\lambda \int \big( f''(t) \big)^2 dt$$
As to why the penalized term takes on this particular form, consider the following: $f''$ is the second derivative of the interpolating function, it represents acceleration/deceleration. In other words, the objective function is penalized by the extent to which the function makes turns and twists. Since we don't care about whether this curvature is positive or negative, we square $f''$. This also penalizes more strongly, the stronger the twist in the function is.
Finally, the purpose of $\lambda$ is to control the amount of regularization. Consider the example figure on Wikipedia's page on regularization:
Here, the blue line could be the minimal loss for $\lambda = 0$ and the green line a penalized version $\lambda > 0$. By choosing a suitable value for $\lambda$, we can smooth the function to avoid overfitting. Methods for choosing $\lambda$ include cross-validation, which has a great explanation in the book you're reading.
