1
$\begingroup$

I have created a simulation in R to fit asymptotic curves (similar to Michaelis-Menten plots) to some univariate data. Essentially, I would like to find the value on the x-axis where curves first reach the asymptote. Sometimes this value is in the range of the X data, other times, it is not, and extrapolation is needed to find the value. For both the interpolation and extrapolation, I am using the package 'investr'.

I want to compare various GAM smooths in mgcv such as bs = "cr" (cubic spline) with bs = "tp" (thin-plate spline). Cubic regression splines seem to be quite popular for univariate smoothing, while thin-plate splines are the default in mgcv due to MSE optimality properties.

Eventually, I want to extend this to shape-constrained additive models (SCAMs) via the R package 'scam'.

On playing around with the different smooths, I am finding that "cr" and "tp" give very similar values. I select the best model with AIC (lowest AIC = best model).

Plots of the GAMs are also very similar, but the AICs are different, which is fully expected. I am starting off with a basis dimension value of k = 20 and increasing k to achieve curve monotonicity.

I have read that choice of k is not overly important as long as it is large enough to avoid over-smoothing but small enough to avoid excessive computation time (time is not an issue in my case).

My question: does it seem reasonable to compare GAMs/SCAMs if they give similar values anyway? For example if "cr" gives a value of x = 30.2 and "tp" gives a value of x = 30.4, it seems rather pointless to choose a "best" model with AIC.

Is my thinking flawed? I am a first-time user of mgcv.

Any advice is greatly appreciated. Thank you.

$\endgroup$
1
$\begingroup$

It's not flawed, per se, just pointless. I can think of (and have experienced) situations where the specific type of basis (of the two you mention) can result in markedly different fits. However, where this has happened to me, it has usually been solved by increasing k for one of the bases or because the wrong model was fitted. In this instances trivial differences in the basis used were magnified by the real problem (needing larger basis dimension, fitting the right model), not because of any fundamental difference in the performance of the individual basis.

In most situations you are going to see trivial differences in the fits of models fitted with different basis (among standard bases) and these are going to result in trivial differences in AIC. In most cases AIC is going to tell you, therefore, that the model fits are equivalent.

If you are planning on using SCAM models, I might suggest you use P splines in the GAMs as the splines in the scam package are all based on P splines.

$\endgroup$
  • $\begingroup$ Thank you Gavin! So, instead of comparing different smooths with GAMs, you're saying it's better to compare a single GAM to a single SCAM, in the univariate case at least? That is compare BETWEEN methods rather than WITHIN methods? I am also using the BFGS optimizer for the GAMs, since it is the one used by the scam package... I see your reasoning in comparing P-splines... it initially crossed my mind to do this. $\endgroup$ – compbiostats Sep 1 '17 at 17:16
  • $\begingroup$ Yes, compare between constrained and unconstrained fits. If you are going to use AIC you need to be very careful that that gam() and scam() are using exactly the same form for the density of the respective distribution you are fitting. If they don't use the same normalising constant then the log-likelihoods of the two models and hence the AIC won't be compatible. A quick glance suggests these two should be comparable, but I'm not that familiar with scam() to say for sure. $\endgroup$ – Reinstate Monica - G. Simpson Sep 1 '17 at 17:24
  • $\begingroup$ The literature does compare between various GAM and SCAM models and even within the same framework (i.e., GAM vs GAM, SCAM vs. SCAM). based on reviewing GAM papers by Simon Wood and SCAM papers by Natalya Pya... Oh and by P-splines you mean use bs = "ps" as the argument in the gam function, which is based on Eilers and Marx (1996)? Because technically all GAMs and SCAMs are P-splines (Penalized splines -- unless using fixed degrees of freedom).. the P-splines of Eilers and Marx use a finite difference approximation of derivatives. $\endgroup$ – compbiostats Sep 1 '17 at 18:31
  • $\begingroup$ Yes, all the splines are subject to penalties in mgcv. But the ps basis is the specific variant or type of penalty suggested by Eilers and Marx. I'm not as familiar with Natalya's work and package but I understood it to be using constrained forms of the ps basis available in mgcv, hence to avoid as many differences unrelated to the thing you are interested in evaluating. As for comparing the models, I have no doubt that people do this; I was just cautioning the implementational differences can render the comparison invalid if you aren't careful. $\endgroup$ – Reinstate Monica - G. Simpson Sep 1 '17 at 18:36
  • 1
    $\begingroup$ Great! Thanks. If I have any other questions, I will post here. It seems that you provide nice answers to a lot of inquiries. $\endgroup$ – compbiostats Sep 1 '17 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.