The event of interest happened only 5 times in the last 4 years. My independent variables are the number of results returned by Google Search for specific keywords over time (and per domain of each result).
I want to check if it possible to predict the event a few days/weeks before it happens (again) which could be shown by increased Google results some days before the event occurs. At least, I want to calculate the probability of the event occurring within the next X days.
My questions are the following:
- Is logistic regression the right way to go for it? I've heard of survival models but have no experience using them. I guess survival analysis is not the proper way to go for it since I do not have many cases (e.g. patients), right?
- If I use LR, how should I format the dependent variable? Now it's an array of around 1.2K 0s (zeros) mixed with five 1s (ones). If I want to predict the probability of the event happening within the next, say, 7 days, should I fill with 1s the 7 places before each existing 1 in the array (hope this makes sense)?
- How do I decide on the length of time-window I use? For my application, anything between 2 and 30 days is fine. Given the class imbalance, I guess using a larger window will help. Should I treat this just as another hyper-parameter of the model (along with regularization parameters)?
- What should I optimize then? Is it better to select model based on log-loss or the f1-score?
- This leads to the question: how do I cross-validate? Currently, I'm using time ordered folds. I'm training on [t0 - t500], then on [t0-t700] and so on and predicting on the left-out observations. Is this the correct approach?