11
$\begingroup$

Can anyone help give a conceptual explanation to how predictions are made for new data when using smooths /splines for a predictive model? For example, given a model created using gamboost in the mboost package in R, with p-splines, how are predictions for new data made? What is used from the training data?

Say that there is a new value of the independent variable x and we want to predict y. Is a formula for spline creation applied to this new data value using the knots or df used when training the model and then the coefficients from the trained model are applied to output the prediction?

Here is an example with R, what is predict doing conceptually to output 899.4139 for the new data mean_radius = 15.99?

#take the data wpbc as example
library(mboost)
data(wpbc)

modNew<-gamboost(mean_area~mean_radius, data = wpbc, baselearner = "bbs", dfbase = 4, family=Gaussian(),control = boost_control(mstop = 5))
test<-data.frame(mean_radius=15.99)
predict(modNew,test)
$\endgroup$
  • $\begingroup$ Question: is this about interpolation (interior to the domain) or extrapolation? They are handled differently. They are also very dependent on "basis" function. A radial basis function like would be expected in a GLM is going to have substantially different behavior, especially out beyond the tails, than some high(er) order polynomial basis. $\endgroup$ – EngrStudent Jul 1 '13 at 19:15
  • $\begingroup$ EngrStudent, I am interested in understanding conceptually what happens in either case. I assumed (maybe incorrectly) that the process was the same in both cases but that the resulting values differ and differ by the basis functions used (but that the process was the same) $\endgroup$ – B_Miner Jul 2 '13 at 12:56
  • $\begingroup$ In polynomial bases there is a phenomena sometimes called "Gibbs effect". If you fit data that is uniform samples of a standard normal distribution to something like a 10th order polynomial, and then look at the quality of interpolation you will see that at the ends the slopes are high and the interpolation is very poor. In polynomial bases it is customary to use a lower order extrapolant than the interpolant. Without knowing the "physics" defining the phenomena the extrapolants are often linear. I use MatLab: mathworks.com/help/matlab/ref/interp1.html . $\endgroup$ – EngrStudent Jul 2 '13 at 13:22
10
+25
$\begingroup$

The way the prediction is computed is like this:

From the original fit, you have knot locations spread through the range of mean_radius in your training data. Together with the degree of the B-spline basis (cubic by default in mboost), these knot locations define the shape of your B-spline basis functions. The default in mboost is to have 20 interior knots, which define 24 cubic B-spline basis functions (don't ask...). Lets call these basis functions $B_j(x); j=1,\dots,24$. The effect of your covariate $x=$``mean_radius`` is represented simply as $$ f(x) = \sum^{24}_j B_j(x) \theta_j $$ This is a very neat trick, because it reduces the hard problem of estimating the unspecified function $f(x)$ to the much simpler problem of estimating linear regression weights $\theta_j$ associated with a collection of synthetic covariates $B_j(x)$.

Prediction then is not that complicated: Given the estimated coefficients $\hat \theta_j$, we need to evaluate the $B_j(\cdot);\; j=1,\dots,24$ for the prediction data $x_{new}$. For that, all we need are the knot locations that define the basis functions for the original data. We then get the predicted values as $$ \hat f(x_{new}) = \sum^{24}_j B_j(x_{new}) \hat\theta_j. $$

Since boosting is an iterative procedure, the estimated coefficients at the stop iteration $m_{stop}$ are actually the sum of the coefficient updates in iterations $1, \dots, m_{stop}$. If you really want to get a grip on the details, take a look at the output you get from

bbs(rnorm(100))$dpp(rep(1,100))$predict,

and go explore from there. For example,

with(environment(bbs(rnorm(100))$dpp(rep(1,100))$predict), newX)

calls

with(environment(bbs(rnorm(100))$dpp(rep(1,100))$predict), Xfun)

to evaluate the $B_j(\cdot)$ on $x_{new}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ this is great. I wonder if you would mind explaining broadly what these functions do? Is it true that what is needed to "score" new data then is the set of coefficients, knot locations used when training and the formula for the splines? Is any other the training data needed to score new data (like in a KNN model)? $\endgroup$ – B_Miner Jul 3 '13 at 15:34
  • 1
    $\begingroup$ Which info you need depends on the type of spline basis you use. For B-splines, all you need to know are the order of the B-splines (quadratic/cubic/etc..) and the knot locations. The "formula" for B-splines is a recursion, the Cox-de Boor recursion. I've added a half sentence to my answer. $\endgroup$ – fabians Jul 10 '13 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.