4
$\begingroup$

The context

I was asked not to give the full context here, but I hope, the following will suffice:

  • There are human caused events usually occurring about weekly.
  • The event as seen by us has no data, just the timestamp. Imagine a user entering some data and finally clicking the "send" button. The click gets reported to us, the data do not.
  • Currently, we're having about 1000 independent data sets.
  • We want to send reminders for expected events.

The data sets are mostly independent as each data set corresponds with an action by a different user (There are some exceptions like one user involved in two or maybe three data sets, but let's ignore them).

We need a prediction of future events, so that we can send a reminder to the user a few hours before they'll produce the next event.

This reminder is to be seen as a service to the users. Some mispredictions are fine, but too many of them would turn the service into a nuisance. In case, no prediction can be done, there'll be no reminder - that's fine.

Real examples

Some real data sets, the format is yymmdd-day_of_the_week-hhmm.

Data set A

  • 180109-Tue-2130
  • 180116-Tue-2259
  • 180124-Wed-1140
  • 180130-Tue-2316
  • 180207-Wed-0105
  • 180213-Tue-2223
  • 180221-Wed-0028
  • 180227-Tue-2116
  • 180307-Wed-0156
  • 180313-Tue-0037

I'd ignore the third and last events as outliers as all others fall in the range Tue-2116 to Wed-0105. Predicting the next event to come on 2018-03-13 at about 23:10 plus minus two hours with a probability of about 80% might be the best guess.

I was told that the last event actually belongs to the others, and that no more event should be expected on 2018-03-14 and this turned out to be correct.

Data set B

  • 171023-Mon-0857
  • 171023-Mon-1618
  • 171109-Thu-2301
  • 171122-Wed-1438
  • 180107-Sun-1452
  • 180131-Wed-1512
  • 180205-Mon-2242
  • 180209-Fri-0040

This looks pretty hopeless. Refusing to generate any prediction sounds best.

Data set C

  • 180204-Sun-2311
  • 180211-Sun-2335
  • 180219-Mon-0110
  • 180226-Mon-0006
  • 180304-Sun-2318
  • 180311-Sun-2208

Predicting the next event on 2018-03-18 at about 23:30 plus minus two hours sounds like a sure bet.

Data set D

  • 180212-Mon-1410
  • 180219-Mon-1205
  • 180226-Mon-1449
  • 180226-Mon-1449
  • 180305-Mon-1834
  • 180312-Mon-2329

This one is fairly regular, too, except for the repeated timestamp on 2018-02-26. Such repetitions are corrections (the use forgot something and added it the next time) and can freely be ignored.

I'd predict the next event on 2018-03-19 at 18:00 plus minus six hours. It might even slide into the next day, but this doesn't make our reminder completely wrong.

I was told, there are data sets containing two events per week, but haven't found such an example yet.

Expected output as computed on 2018-03-13 at 01:00

  • For data set A (my original thought):
    Next event on 2018-03-13 at about 23:10 plus minus two hours with a probability of about 80%.
    => Produce a corresponding reminder.

  • For data set A (improved):
    Next event on 2018-03-20 at about 23:10 plus minus two hours with a probability of about 80%.
    => No reminder for now as it's too distant future (the computation will be repeated at least daily).

  • For data set B:
    Unpredictable.
    => No reminder.

  • For data set C:
    Next event on 2018-03-18 at about 23:30 plus minus two hours with a probability of about 90%.
    => A reminder.

When we predict a next event in the next few hours with some sufficient probability, we'll send a reminder to the user. Currently, we plan to send the reminder 6 hours before the expected time and require a probability of at least 60%, but this will change according to the feedback we get.

Clarifications to the comments

We get all the events immediately as they get produced. The above examples were produced from a state as of 2018-03-03 01:00 and you can obtain this state from the linked file by simply removing all later events.

Obviously, you can generate our data at any older time t by simply removing all events newer than t.

At any time t, our data contain all events before t (especially the first event). Nonetheless, we believe than only a few (maybe 10) most recent events are relevant for the prediction (it's just a gut feeling).

There may be correlations between the data sets, but they're probably too minor to matter in our limited data. There'd be useful, if we knew the future of some data sets, but we obviously don't.

The events may be influenced by major TV events and by holidays, but IMHO that would be visible with much more data sets only. We most probably can ignore all external events as they fall below the noise level.

The question

How can I produce such predictions?

I'm linking a file containing about 1000 data sets from 2018-03-25 at about 03:30: https://www.dropbox.com/s/ukwqlqxb0dgzq6g/stats-334010.txt?dl=0

Data sets containing fewer than three events were removed as uninteresting as we need no prediction for them.

Improve this question
$\endgroup$
24
  • 4
    $\begingroup$ What is your question? Eg, I don't see a "?" anywhere. $\endgroup$
    – gung - Reinstate Monica
    Mar 19, 2018 at 13:36
  • 2
    $\begingroup$ Given that a big database is presented and also the intentions/goals are made very clear ("We need a prediction of future events, so that we can send a reminder to the user a few hours before they'll produce the next event.") I would say to open this question. It might be considered still broad or doing basic development work, but in that case the question should have been deleted long before. I believe that there is at least some value to answer this question and the OP seems to be willing to adjust the question if it would be useful to make it more generic than the current specific setting $\endgroup$
    – Sextus Empiricus
    Mar 25, 2018 at 20:39
  • 1
    $\begingroup$ @MartijnWeterings I acknowledge, you we're arguing in favor of it. I guess, I'll be able to customize the approach myself, it's just that I'd like to get an advice about what makes most sense. I'm not looking for an ultimate solution - more data and more requirement will come later (in a few months maybe) and something better may be developed. +++ Your translation to an SO problem sounds very broad to me. I'd translate it like "given a set of points, find the "smallest" set of rectangles containing them all + some explanation how "smallest" is to be understood and what it's good for). $\endgroup$
    – maaartinus
    Mar 25, 2018 at 21:27
  • 1
    $\begingroup$ IMHO, this has turned into an interesting well-framed question (+1). Thank you for continuing to work on it. $\endgroup$
    – whuber ♦
    Mar 25, 2018 at 21:35
  • 3
    $\begingroup$ The data have to be censored. There is a timepoint when the datasets were generated. These users can still produce events after that, but haven't yet. That's the definition of censoring. $\endgroup$
    – gung - Reinstate Monica
    Mar 26, 2018 at 14:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.