6
$\begingroup$

I am trying to figure out how to explain the apparent discrepancy between P-values and plotted confidence intervals in mgcv.

See for example the plot below that comes from a model that considers the association between a serum marker of diabetes and pregnancy complications. The P-value for the smooth term based on the GAM output is 0.01, but the partial effect plot clearly shows a confidence band that spans the null value of zero.

Couple of questions:

  1. I read that in the mgcv implementation of GAM, computation of p-values for the smooth terms does not take into account uncertainty of the smoothness parameter. What does that mean exactly? In practical terms, my understanding is that this means that the observed P-values might be too low compared to where they should be.

  2. Relatedly, for plotting, using the seWithMean option in the plotting allows one to show the component smooths with confidence intervals that include the uncertainty about the overall mean – which should result in better coverage performance. However, choosing this does not seem to affect the P-value calculations.Correct? But shouldn't the P-value take into account all uncertaintly for the term in the model?

  3. I am unclear about the use of the ‘unconditional’ option. The vignette suggests that if set to TRUE, the smoothing parameter uncertainty corrected covariance matrix is used to compute uncertainty bands, if available. Otherwise the bands treat the smoothing parameters as fixed. What does it mean “if available”? and what does it all mean in general?

  4. Finally, for the bottom line – when using splines, we often use plotting as the primary mean of showing the result. However, people often request a p-value as well. When the P-value doesn’t agree with the plots, what does one do?

Thank you!

enter image description here

$\endgroup$
1
  • $\begingroup$ Is there a particular reason for why the standard error looks this way? $\endgroup$ Oct 21, 2023 at 0:39

1 Answer 1

5
$\begingroup$

Q1

The theory used to compute the p value of the test of the null hypothesis of a flat, constant function equal to 0 treats the smoothing parameters (the things that ultimately determine the wiggliness of the smooth functions) as fixed and known. However, these were selected by mgcv during fitting. Hence the p value doesn't account for this extra source of uncertainty, and are, as a consequence a little too liberal, a little too anti-conservative, a little too approximate. And that last point is on top of the approximation for p values in generalized linear models (which apart from the Gaussian, rely on large sample approximations).

That said, you really shouldn't be making decisions on the basis of p values in applied science work, the estimated function and effect sizes are more important.

Q2

Correct; selecting seWithMean = TRUE will not change the computation of the p value via summary(), nor can it; these are two separate functions. The issue that seWithMean option is trying to correct is visible when smooths are estimated to be linear functions. Then, because of the sum-to-zero constraint applied to all the smooths (which is what allows a constant term — the intercept — in the model), the smooth function must pass through the point $\widehat{f(x)} = 0$. At that point, there is no uncertainty in the value of the smooth, by definition the value is 0 with no uncertainty about that.

From the point of view of creating an interval about the smooth that has appropriate coverage (95% coverage if a 1-0.05 interval is formed), this behaviour is problematic; there is no way that a zero-width confidence interval makes any sense (which is what one would get if one used the standard error of the smooth = 0 at this zero-crossing point) because it implies that the smooth must pass through this point.

What seWithMean does is apply the uncertainty in the model constant term to each smooth in turn while forming the interval. This adds the uncertainty in the vertical position to the uncertainty in the smooth function itself. It turns out that this correction provides an interval with the correct coverage, except for the situation where the true function is nonlinear but close to the penalty null space (i.e. the true but unknown function is close to linear).

It's not clear to me that this extra uncertainty should be used in computing a p value. That p value is for a test of a specific hypothesis, which isn't affected by the vertical position of the smooth. Hence, from the point of the test of that very specific null hypothesis (i.e. a null of a flat constant function), this extra uncertainty that is needed to make an interval have the right coverage is not necessary when formulating the Wald-like test against the flat function.

Q3

The theory needed to correct the covariance matrix for smoothness parameter selection requires the calculation of certain values and those are only done if we fit the model using specific smoothness selection methods.

When we fit these GAMs, we are fitting a penalized likelihood model, where we take into account the wiggliness of the estimated smooths. This wiggliness is a penalty; the wigglier the smooth the more complex the function. We use smoothing parameters to control how much price we pay for the wiggliness penalty. If the smoothing parameter is large we pay a heavy penalty for wiggliness and so we'll tend to pick very smooth functions. If we have a small smoothing parameter we pay less penalty for wiggliness and hence we can fit wigglier, more complex functions.

The only problem is that when we fit the model we don't know what the values of these smoothing parameters are, and instead estimate them during fitting, but when we compute tests and other statistical values we assume we knew the values of these smoothing parameters all along. Clearly, not knowing the values of these smoothing parameters but acting as if we did know them, would mean we assume our uncertainties are smaller than what they actually are.

The unconditional argument is a way to correct, to an extent, for the fact that we did not know the values of these smoothing parameters after all. But the required correction can only be computed for some approaches to smoothness selection. Typically this is when method = "REML" or method = "ML". The default is method = "GCV" and with that method the correction is not available.

Q4

Fundamentally, the credible interval and the p value aren't intended to convey or reflect or test the same quantity of the smooth.

What I think is happening in this plot is that we are reasonably confident that the function is not linear and not a constant 0 function, but we are unsure exactly where the non-linear function lies. This latter point is reflected by the width of the interval. Just because you can draw a straight line through the interval doesn't mean a constant function is consistent with the estimated function.

$\endgroup$
8
  • 1
    $\begingroup$ Thanks as always Gavin for the informative response. I am still trying to develop an intuition for this. Putting aside their limitations as you indicated, in a conventional GLM, the p-value for each term still provides some information about the strength of evidence for an association with the outcome. So if I create a plot of a penalized smooth term that shows some wiggleness but with confidence bands that span the null value of zero, wouldn’t you expect the P-value to be nonsignificant? Can you elaborate on why the credible interval and the p value convey different information? $\endgroup$
    – dean
    Oct 22, 2023 at 19:19
  • $\begingroup$ Let me attempt answering my own question: The P-value provides some indication of an association with the outcome. The edf suggest that this association is also non-linear. The confidence bands (with seWithMean=T) reflect uncertainty about the smooth term + uncertainty about the model constant. This suggests that the confidence bands can span the null value of zero even if the function itself is not a constant, b/c of uncertainty around the constant. Would that also suggest that plotting with seWithMean=F would more closely reflect the P-value in terms of the bands excluding the null value? $\endgroup$
    – dean
    Oct 22, 2023 at 19:53
  • 1
    $\begingroup$ The p tests a specific null hypothesis of a flat (constant) function. In your case, the test reflects the weight of evidence in favour of this specific hypothesis. The estimated function seems inconsistent with a flat function; the data suggest a non-linear functional relationship between the response and the covariate. Put simply, the estimated function is incompatible with the null hypothesis of a flat function, and that is why the p value is small despite the credibly interval spanning 0 throughout. $\endgroup$ Oct 22, 2023 at 21:01
  • $\begingroup$ If that plot is with seWithMean = TRUE then you aren't really seeing an interval that matches with what the p value is testing. Are things any clearer if you set this option to FALSE, is 0 still in the interval everywhere? If you were to plot posterior draws from the estimated function if might also help understand what the interval is there to convey versus what the p value is doing/testing. $\endgroup$ Oct 22, 2023 at 21:03
  • $\begingroup$ Thanks! so going back to the last point, it seems like plotting with seWithMean = FLASE matches closer with what the P value is testing, correct? I know that doing this is unwarranted due to lower coverage performance of the confidence bands, but just to make sure I understand the diffrerence. And is theere a straightforward way to plot posterior draws from the estimated function? Thank you! $\endgroup$
    – dean
    Oct 23, 2023 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.