I have a data set with only a few predictor variables (2-3) but with millions of observations. These data represent daily physical conditions for juvenile fish travelling across the ocean for a few years years, before they recruit into the fishery. The response variable is recruitment, which is just about 30 annual values, in number of individuals being added to the adult population. So, the problem that I’m running into is that I have an awesome data set of about 2-3 predictor variables with a huge number of observations (millions), that I need to use to model a response variable with very few number of observations (about 30).
I have already summarized by year and modelled recruitment with a mixed model and a GAM, but I'm just losing so much information in doing that. Going from millions of data points down to 30... I know I'll have to summarize the data in some way first because it's so large, but still feels bad.
Is there some way that I can use the large number of observations of the predictor variables to model the small number of observations of the response variable? A time series analysis, or bootstrapping maybe?
