# Centering constraints for regression - specifically GAM

I asked the following question and (don't think) really received a thorough answer. I qualify my impression about the suitability of the answer, because perhaps I am lacking pre-requisite knowledge.

I. In ordinary linear regression without nominal covariates, i.e. $y= b_o + b_1 \times x_1 + b_2 \times x_2 + e$ where $x_1$ and $x_2$ are numeric covariates, is there any constraints on $b_1$ and $b_2$?

(I know that the X design matrix will not be invertible when there are "dummy variables" and more than 2 levels of a given nominal predictors - hence I mention only numeric covariates here).

If there is no constraints on the coefficients $b_1$ and $b_2$ then why are there constraints on $f_1(x)$ and $f_2(x)$ in a GAM? From Simon Wood's text:

II. Are these constraints required when considering a generalized linear | additive model?

• Your Q title mentions centring constraints but then you discuss identifiability constraints in the body. Are you interested in both or just the latter? Aug 6, 2012 at 22:43
• Both, I am apparently using the term centering constraint incorrectly. Aug 7, 2012 at 2:21
• I don't think so, I just wanted to know if you were specifically interested in centring constraints or not. I've had another stab at answering this from the GAM point of view. Aug 7, 2012 at 10:57

Identifiability constraints and centring constraints are two related things. A little further on in Wood (2006), Simon goes on to mention that the $f_j(x)$, $j = {1, 2}$, can be represented as a penalised regression spline basis. (All of the models in mgcv's gam() are fitted using penalised regression.) Simon then shows the two equations for $f_j(x)$ using a spline basis from section 3.2.1 are

$$f_1(x) = \delta_1 + x\delta_2 + \sum_{j=1}^{q_1-2} R(x,x_j^*)\delta_{j+2}$$

and

$$f_2(z) = \gamma_1 + z\gamma_2 + \sum_{j=1}^{q_2-2} R(z,z_j^*)\gamma_{j+2}$$

where $\delta_j$ and $\gamma_j$ are the unknown parameters and $q_1$ and $q_2$ record the number of unknown parameters for $f_1$ and $f_2$ respectively. $x_j^*$ and $z_j^*$ are the locations of the knots for the two smooth functions.

In order for us to learn the true values of $\delta_j$ and $\gamma_j$ the model needs to be identifiable. Simon's comment in the text you quote says that the $f_j$ are not fully identifiable; we can only estimate their true values up to the additive constant he mentions. If we can't identify the true values of the smooths then they aren't much use for telling us something about our data and the population from which the data are drawn. It would be like saying the estimated values of the $b_j$ in your linear model could take any value; how would you interpret the estimated coefficients if they could take any value? You couldn't.

The $f_j$ can be made identifiable by setting one of $\delta_1$ or $\gamma_1$ to be zero. In other words the value of the coefficient $\delta_1$ or $\gamma_1$ is constrained to be zero. With that the $f_j$ can be uniquely identified and that allows us to estimate their true values.

Elsewhere [pdf] Simon describes the identifiability constraints more generally as "that each smooth should sum to zero over its covariate values" (page 8 in the preprint). In my head this is a centring constraint as the smooths are centred about zero. And this makes the link nicely to the centring constraints in by terms.

As for the centring constraints in by smooths, each separate smoother is constrained so that it sums to zero over the values of the covariate; this allows the smooths to be identified as per the above. As the smooths each sum to zero, the smoothers can only represent variation in $y$ about a common mean. In other words, the $f_{g}$ describes how the response varies as a function of $x$ for each group $g$ about a common mean value. As such a model including $f_{g}(x)$ (a by smooth) allows for different shaped relationships between $x$ and the response $y$, but does not include any systematic difference in the mean of $y$ in the different $g$ groups.

That is the point I was trying to make in the Q&A you linked to; the reason you need to include the factor by variable as a parametric term in the model as well as the by smoother is to allow the model to capture any differences in the means of $y$ over the various groups. That variance would not be accounted for if you just include the by smoother.

• (+1) that response helped me feel better about understand generally what is happening - thank you! Is it your opinion that each $f_{g}(x)$ is smoothing basically the subset of {y,x} where the by variable equals level g? You confused me a little when mentioning "for each group g about a common mean" - unless maybe that common mean is zero. If that is true then I get it I think. The issue is each smooth describes how y varies around zero, as a function of x (different for each level of by variable), but we still don't include how the mean of y varies between levels of the by variable. Aug 7, 2012 at 12:29
• Depends where you look. At the level of the $f$, then the common mean is zero, but remember the entire model has an intercept (constant) term which combines additively with the $f$. So at the level of the model the $f$ describe how $y$ varies with $x$ within the groups about some non-zero common mean; the non-zero bit coming from the value that the intercept takes. Aug 7, 2012 at 12:39
• On the other bit; in by terms, the model matrix gets a set of indicator variables that indicate if an individual $f_g$ applies to a row in the data. If the indicator is 0, that particular smooth does not apply to that row of the data and hence more generally yes it $f_g$ basically applies to the subset of {y,x} where by is the $g$th level. I say basically as I can't remember if the basis functions for each $f_g$ are the same for each by level or if a different set of basis functions is used for each level. Aug 7, 2012 at 12:43
• Hi @Gavin Simpson, I added a question related to this one, that you may have insight into, given that you appear to really be the GAM expert here. If you get a chance / have interest, it is here: stats.stackexchange.com/questions/113107/… Aug 25, 2014 at 2:43