I am reading ISL and found that fitting a cubic spline with $K$ knots uses $K+4$ degrees of freedom, since it estimates $K+4$ regression coefficients (p.273). However, in ESL they say that a natural cubic spline with $K$ knots is represented by $K$ basis functions (p.145), however according to ISL (p.275) these are $K+1$ degrees of freedom. My understanding is that since it requires fitting $K$ basis functions (the first of which is the intercept) it should also use $K$ degrees of freedom. What am I missing here?
-
$\begingroup$ Did you read the footnote on page 275 of ISLR? $\endgroup$– angryavianOct 3, 2020 at 23:46
-
$\begingroup$ I did yes, and it does make sense. But if you think of the $K$ final basis functions that you fit (with no additional constraints), I do not see where that extra parameter comes from. $\endgroup$– econ86Oct 4, 2020 at 0:01
1 Answer
$K$ is then number of knots, including the exterior knots (beyond which the natural splines are constrained to be linear). The example on pages 274-5 of ISLR puts three knots on the 25th, 50th, and 75th percentiles, but also two more knots on the boundaries of the data. [This was not explicitly explained in the body of the text, but it was mentioned in the footnote; you can also infer this based on the fact that the left panel of Figure 7.5 is not linear below the 25th percentile or above the 75th percentile.] So in that example, $K=5$, not $3$. The footnote then goes on to say that this "results in $9-4=5$ degrees of freedom" which equals $K$ and does not contradict ESL. (I am not sure why the last sentence of the footnote concludes with $4$; for some reason they removed the degree of freedom corresponding to the intercept, which contradicts the convention they used when counting $K+4$ degrees of freedom in the previous section on cubic splines.)
Response to comment:
$K$ counts all knots, including the two boundary knots. If you insist on counting interior knots, perhaps it would help make the distinction clearer by introducing the notation $K_{\text{interior}} = K-2$. The claim then is a natural spline with $K$ knots (or $K_{\text{interior}} = K-2$ interior knots) has $K = K_{\text{interior}}+2$ degrees of freedom. This is what ESL claims; the example in ISL has $K=5$ and the footnote states "this results in $5$ degrees of freedom." There is no "$K+1$" in ISL; if anything, that example is somehow resulting in $K-1$ because they inexplicably throw away the intercept degree of freedom.
-
$\begingroup$ Thanks. Part of my confusion came from the change in convention you mentioned. But also one point remains: if $K$ interior knots require $K+1$ degrees of freedom but only fit $K$ parameters, where is that extra parameter coming from? $\endgroup$– econ86Oct 4, 2020 at 8:44
-