I was suggested that my questions were too broad. As I commented below, I have nearly a million data points and perhaps a hundred variables. This may be a very basic modeling question: I am curious to know how to start a GAM with a large dataset. I have tried the 'bam' function with a much smaller dataset, and it didn't work as what I expected. I do have access to supercomputers, but it still seems unpractical to tune a GAM with this big dataset. I was suggested to pick 8 to 10 variables and fit a GAM. Still, it is slow to run a GAM with the complete dataset. So my guess is that I need to reduce the number of variables and sample size to fit a GAM.

My original questions: I have 61 bioclimatic variables that explain different or similar aspects of insect life cycles and some of them are highly correlated. My study extent covers the North American continent and the spatial resolution is 10 km. The temporal resolution is yearly and temporal range is 20 years. This means that my dataset is huge for GAMs. I have built models using GLMs instead for prediction purpose. However, the models are complicated (e.g., 777265*263) and not easy to interpret. So I am trying to use GAMs to build small models that only include fewer variables and some percentage of samples for interpretation purpose. I followed some questions on the package 'mgcv' and found that most of the examples are using a very small number of variables. Does that mean I need to handpick the variables? I used the 'gam.selection' function with a smaller dataset (828*54), and I can see some variables are not significant in a smoothed term. I also used the 'concurvity' function to examine potential multicollinearity. Now I need some suggestions on variable selection: What are the appropriate number of variables for an explanatory GAM? Do I select the variables based on my knowledge, the 'gam.selection' results where a significant nonlinearity is detected, and the 'concurvity' results? Or what would be the most efficient way in this variable selection process? I appreciate your thoughts and timely help.

  • $\begingroup$ You should probably explain what some of these numbers mean; is 263 just the number of covariates before turning them into smooths? For example... If you really don't know which smooth effects to retain, I would suggest turning on select = TRUE and using the bam() function in mgcv. $\endgroup$ – Gavin Simpson May 7 '19 at 19:26
  • $\begingroup$ I've written at least one answer on this general area: stats.stackexchange.com/a/405292/1390 $\endgroup$ – Gavin Simpson May 7 '19 at 19:36
  • $\begingroup$ Thanks for replying! The numbers above were mostly to show my data properties and you can skip them if you get the idea that I have a very large sample size (nearly 1M) and a long list of variables. I have tried both gam.selection and bam functions on a smaller dataset (surprisingly, bam is slower than gam in my case; but I may have used wrong parameters). My questions were how to tune GAM for my data at the beginning and what to do next after gam selection (for example, do I remove all the variables that are not significant after selection and tune the smooth effects with the rest variables?) $\endgroup$ – Dongmei Chen May 7 '19 at 20:57
  • $\begingroup$ Have you tried s(..., bs='cr') within gam (not bam)? $\endgroup$ – usεr11852 May 7 '19 at 22:16
  • $\begingroup$ @usεr11852 That will save you time on constructing the basis, but computationally with ~ 0.75 million observations the bottleneck is going to be the data size. bam() with s(..., bs = 'cr') is probably as efficient it is going to get with the sorts of models mgcv can fit. $\endgroup$ – Gavin Simpson May 7 '19 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.