10
$\begingroup$

Suppose you have the logs of a web server. In these logs you have tuples of this kind:

user1, timestamp1
user1, timestamp2
user1, timestamp3
user2, timestamp4
user1, timestamp5
...

These timestamps represent e.g. users' clicks. Now, user1 will visit the site multiple times (sessions) during the month, and you'll have bursts of clicks from each user during each session (supposing that when a user visits your site, he'll click on multiple pages).

Suppose you want to partition these burst of clicks in the sessions that generated them, but you don't have any additional source of information, only the list of timestamps. If you compute the distribution of intervals between two consequent clicks from the same user, you will obtain a long-tailed distribution. Intuitively, you'd look for a "cut parameter", e.g. N seconds, where if timestamp_{i+1} - timestamp{i} > N, then your timestamp_{i+1} is the beginning of the new session.

The problem is that this distribution in reality is a mixture of two variables: X = "interval between two consequent clicks in the same session" and Y = "interval between the last click of the previous session and the first of the new one".

The question is, how to estimate this N, that divides the two distributions (with a bit of overlap, possibly) just by looking at the burst of clicks?

Improve this question
$\endgroup$
3
  • $\begingroup$ When you say "just by looking at the burst of clicks" do you mean you're unable to compute anything other than N? $\endgroup$
    – jerad
    Nov 23, 2012 at 4:18
  • $\begingroup$ I mean that you don't have any additional sources of information other than the tuples (user, timestamp). The threshold-based method (based on delta > N) is just an example of a method. Maybe something else is possible. $\endgroup$
    – marcorossi
    Nov 23, 2012 at 15:51
  • $\begingroup$ This thread may be of interest to you: appropriate-clustering-techniques-for-temporal-data. $\endgroup$
    – gung - Reinstate Monica
    Nov 28, 2012 at 16:23

1 Answer 1

Reset to default
2
$\begingroup$

You really should plot the logarithm of the inter-click intervals instead of the raw values; this will flatten your distribution and might even reveal the multiple modes in your distribution.

More advanced approaches have been developed by neuroscientists to solve a very similar problem in identifying bursts of neuronal spikes. This classic paper or the many other related papers on google scholar.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I did print the loglog of the distribution. It's a flat line. How does that help though? What would you look at? The reference for the paper is great, thanks. $\endgroup$
    – marcorossi
    Dec 14, 2012 at 14:39
  • $\begingroup$ What about just the log probability plot? i.e. take the log of only the frequencies, not the intervals. Does that reveal two modes? $\endgroup$
    – jerad
    Dec 14, 2012 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.