The event of interest happened only 5 times in the last 4 years. My independent variables are the number of results returned by Google Search for specific keywords over time (and per domain of each result).

I want to check if it possible to predict the event a few days/weeks before it happens (again) which could be shown by increased Google results some days before the event occurs. At least, I want to calculate the probability of the event occurring within the next X days.

My questions are the following:

  1. Is logistic regression the right way to go for it? I've heard of survival models but have no experience using them. I guess survival analysis is not the proper way to go for it since I do not have many cases (e.g. patients), right?
  2. If I use LR, how should I format the dependent variable? Now it's an array of around 1.2K 0s (zeros) mixed with five 1s (ones). If I want to predict the probability of the event happening within the next, say, 7 days, should I fill with 1s the 7 places before each existing 1 in the array (hope this makes sense)?
  3. How do I decide on the length of time-window I use? For my application, anything between 2 and 30 days is fine. Given the class imbalance, I guess using a larger window will help. Should I treat this just as another hyper-parameter of the model (along with regularization parameters)?
  4. What should I optimize then? Is it better to select model based on log-loss or the f1-score?
  5. This leads to the question: how do I cross-validate? Currently, I'm using time ordered folds. I'm training on [t0 - t500], then on [t0-t700] and so on and predicting on the left-out observations. Is this the correct approach?
  • $\begingroup$ Check en.wikipedia.org/wiki/Extreme_value_theory $\endgroup$ – Tim Jun 27 '16 at 13:15
  • $\begingroup$ Are you predicting event-arrival-time as a function of any other parameters? Are they themselves time-series in nature? Or are you just considering time itself to be the only predictor (like in reliability analysis)? $\endgroup$ – AdamO Jun 27 '16 at 14:32
  • $\begingroup$ @AdamO As I wrote above, the independent variables are the number of results returned by Google Search for specific keywords over time (i.e. time-series themselves). So some features are 'total results', 'total results from wikipedia.org' etc. on which I'm calculating rolling Max, Avg, Sum, Variance. This leads to a total of around 6K features. $\endgroup$ – Stergios Jun 27 '16 at 15:44
  • $\begingroup$ @Stergios you simply are not powered to detect anything usable. Any such analyses would be completely exploratory. As a type of exploratory analysis, you can simply inspect which features tended to precede the event of interest by a relatively short interval of time. Logistic regression is also powered by the number of events, and 5 is too small for any purpose. $\endgroup$ – AdamO Jun 27 '16 at 16:07
  • $\begingroup$ @AdamO If I format the dependent variable according to my 2nd point, then the 5 events become 150 (with a window size of 30 days). 150 is around 1% of my total points, so I'd expect that I could make something usable, no? $\endgroup$ – Stergios Jun 28 '16 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.