I want to build a predictive model of an event in the spring based off of the weather during the winter (variable every year) and the soil characteristics (fixed) of many different sites.
Although I started with a fairly big dataset, by the time I averaged the values over the year (for 1 "event" value per year) and got rid of some of the sites that didn't fit the right profile, I only have 26 response variable values in the training set.
Looking at the weather data, there are many ways I can define what happened with the weather in the winter - how cold (lowest temp, mean min temp, frequency of days under -x degrees, etc), how warm (similar), how wet (mean precip, cumulative precip, etc.). The soil characteristics are more straightforward, but I can choose how deep I draw these characteristics from.
It was recommended I start with linear stepwise regression and "play around" with models until I find a good fit. So I just chose a few (non-correclated) predictor variables that made the most scientific sense, wrote a simple linear model, stepwise kicked almost everything out, and that was that, one variable explained everything. I also ran the same model through LOOCV and got different results. None of the fits are great, but I don't expect them to be.
If you consider the different weather and soils variables I can come up with, plus transformations and/or interactions of these, plus the order I list them for stepwise regression, I could be "playing around" with models forever.
I started reading Applied Predictive Modeling and read about PLS and it sounds really good. What I really like is that it deals well with correlated predictors. What I am thinking about doing is creating as many predictor variables as I can think of and running them through PLS to find which variables explain the response the most. Is that really how it works?
Then would I take these top predictor variables and use them to write a simple linear regression model that I would test on my validation set???
It just feels like there are so many things I could do here, but writing out hundreds of models by hand and testing each one does not seem like one of them.
I am doing all of this in R.