Presence/Absence GAMM with penalized smoother convergence/completion problem, data set too large

Question

I am having issues running a GAMM to determine presence/absence with a very large data set (9,000,000 rows x 13 cols) using GAM4 in R. I think the data set may be too large and is causing convergence/completion issues. Does anyone have insight to make the process more efficient or if there is an alternative method. It seems the most appropriate method is to implement a GAMM with a penalized smoother as forward and backward step-selection could introduce bias. I have attempted to run a GAMM with a “ts” smoother on 4 variables, a “cc” smoother for day of year (yd) and 2 other variables that are factors with no smoother, and Individual (Transmitter) as a random factor using this command.

gammodelI <- gamm4(P_A ~ s(sst, bs = "ts") + s(a_443_qaa, bs = "ts") + 
                   s(a_555_qaa, bs = "ts")+ X9.Pops_12.loci + 
                   M_WK+s(Depth,bs = "ts") + s(yd, bs = "cc"),
                   family = "binomial", data = data, random =  ~ (1|Transmitter))

A sample dataset (~10%) can be found here: http://modata.ceoe.udel.edu/dev/mwbreece/sample_data_cross_val/
The data is a little different than typical presence/absence data so I included a brief description below.

In the end I would like to make a predictive model of presence and absence using a few environmental variables that I can project onto a larger area than just the 100 receiver locations. Any insight into making one or more of the steps along the way more efficient or corrections to my approach is very much appreciated, I have been trying to solve this problem for a good bit now.
Thanks in advance, Matt

Data description: My data is acoustic telemetry data from an underwater passive acoustic array with 18 associated daily environmental data variables (temperature, depth, amount of chlorophyll, etc.) at 100 acoustic receiver locations over a 4 year period. I have ~ 260 transmitter fish that I know the unique ID of and need to account for individual effect and hence the mixed model. The data for one individual (Transmitter ID 001) on a given day would be – fish (Transmitter ID 001) was present on Receiver X and the environmental data is y, z, … 18 variables. So that is one row with presence being 1, unique Transmitter ID 001 and then the environmental variables. For the same day I have 99 additional rows with presence = 0, Unique transmitter ID 001 and the environmental variables (this is all of the locations the at the fish was absent). This is for one individual on one day, and I have this repeated for up to 4 years just for one individual. This is then repeated for the other 259 individuals with unique transmitter IDs. All receivers and all transmitters were not deployed continuously for the entire 4-year period and this has been accounted for in the number of rows for each individual. I have narrowed the number of environmental variables down to 7 using information value and correlation.

Answer 1

I would try running this using the bam() function in the mgcv which is designed for large data problems. You can add the simple random effects via a random effect spline s(transmitter, bs = "re") in the bam() formula, but this might cause you new problems as random effects in gam()/bam() don't make use of the sparsity of the model matrix of the random effects. If that causes a problem, you can also use the paraPen argument to include random effects but that may suffer the same issue as the random effects as spline problem.