I was asked not to give the full context here, but I hope, the following will suffice:
- There are human caused events usually occurring about weekly.
- The event as seen by us has no data, just the timestamp. Imagine a user entering some data and finally clicking the "send" button. The click gets reported to us, the data do not.
- Currently, we're having about 1000 independent data sets.
- We want to send reminders for expected events.
The data sets are mostly independent as each data set corresponds with an action by a different user (There are some exceptions like one user involved in two or maybe three data sets, but let's ignore them).
We need a prediction of future events, so that we can send a reminder to the user a few hours before they'll produce the next event.
This reminder is to be seen as a service to the users. Some mispredictions are fine, but too many of them would turn the service into a nuisance. In case, no prediction can be done, there'll be no reminder - that's fine.
Some real data sets, the format is
Data set A
I'd ignore the third and last events as outliers as all others fall in the range Tue-2116 to Wed-0105. Predicting the next event to come on 2018-03-13 at about 23:10 plus minus two hours with a probability of about 80% might be the best guess.
I was told that the last event actually belongs to the others, and that no more event should be expected on 2018-03-14 and this turned out to be correct.
Data set B
This looks pretty hopeless. Refusing to generate any prediction sounds best.
Data set C
Predicting the next event on 2018-03-18 at about 23:30 plus minus two hours sounds like a sure bet.
Data set D
This one is fairly regular, too, except for the repeated timestamp on 2018-02-26. Such repetitions are corrections (the use forgot something and added it the next time) and can freely be ignored.
I'd predict the next event on 2018-03-19 at 18:00 plus minus six hours. It might even slide into the next day, but this doesn't make our reminder completely wrong.
I was told, there are data sets containing two events per week, but haven't found such an example yet.
Expected output as computed on 2018-03-13 at 01:00
For data set A (my original thought):
Next event on 2018-03-13 at about 23:10 plus minus two hours with a probability of about 80%.
=> Produce a corresponding reminder.
For data set A (improved):
Next event on 2018-03-20 at about 23:10 plus minus two hours with a probability of about 80%.
=> No reminder for now as it's too distant future (the computation will be repeated at least daily).
For data set B:
=> No reminder.
For data set C:
Next event on 2018-03-18 at about 23:30 plus minus two hours with a probability of about 90%.
=> A reminder.
When we predict a next event in the next few hours with some sufficient probability, we'll send a reminder to the user. Currently, we plan to send the reminder 6 hours before the expected time and require a probability of at least 60%, but this will change according to the feedback we get.
Clarifications to the comments
We get all the events immediately as they get produced. The above examples were produced from a state as of 2018-03-03 01:00 and you can obtain this state from the linked file by simply removing all later events.
Obviously, you can generate our data at any older time
t by simply removing all events newer than
At any time
t, our data contain all events before
t (especially the first event). Nonetheless, we believe than only a few (maybe 10) most recent events are relevant for the prediction (it's just a gut feeling).
There may be correlations between the data sets, but they're probably too minor to matter in our limited data. There'd be useful, if we knew the future of some data sets, but we obviously don't.
The events may be influenced by major TV events and by holidays, but IMHO that would be visible with much more data sets only. We most probably can ignore all external events as they fall below the noise level.
How can I produce such predictions?
I'm linking a file containing about 1000 data sets from 2018-03-25 at about 03:30: https://www.dropbox.com/s/ukwqlqxb0dgzq6g/stats-334010.txt?dl=0
Data sets containing fewer than three events were removed as uninteresting as we need no prediction for them.