Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Suppose you have the logs of a web server. In these logs you have tuples of this kind:

user1, timestamp1
user1, timestamp2
user1, timestamp3
user2, timestamp4
user1, timestamp5
...

These timestamps represent e.g. users' clicks. Now, user1 will visit the site multiple times (sessions) during the month, and you'll have bursts of clicks from each user during each session (supposing that when a user visits your site, he'll click on multiple pages).

Suppose you want to partition these burst of clicks in the sessions that generated them, but you don't have any additional source of information, only the list of timestamps. If you compute the distribution of intervals between two consequent clicks from the same user, you will obtain a long-tailed distribution. Intuitively, you'd look for a "cut parameter", e.g. N seconds, where if timestamp_{i+1} - timestamp{i} > N, then your timestamp_{i+1} is the beginning of the new session.

The problem is that this distribution in reality is a mixture of two variables: X = "interval between two consequent clicks in the same session" and Y = "interval between the last click of the previous session and the first of the new one".

The question is, how to estimate this N, that divides the two distributions (with a bit of overlap, possibly) just by looking at the burst of clicks?

share|improve this question
    
When you say "just by looking at the burst of clicks" do you mean you're unable to compute anything other than N? –  jerad Nov 23 '12 at 4:18
    
I mean that you don't have any additional sources of information other than the tuples (user, timestamp). The threshold-based method (based on delta > N) is just an example of a method. Maybe something else is possible. –  marcorossi Nov 23 '12 at 15:51
    
This thread may be of interest to you: appropriate-clustering-techniques-for-temporal-data. –  gung Nov 28 '12 at 16:23
add comment

1 Answer

You really should plot the logarithm of the inter-click intervals instead of the raw values; this will flatten your distribution and might even reveal the multiple modes in your distribution.

More advanced approaches have been developed by neuroscientists to solve a very similar problem in identifying bursts of neuronal spikes. This classic paper or the many other related papers on google scholar.

share|improve this answer
    
I did print the loglog of the distribution. It's a flat line. How does that help though? What would you look at? The reference for the paper is great, thanks. –  marcorossi Dec 14 '12 at 14:39
    
What about just the log probability plot? i.e. take the log of only the frequencies, not the intervals. Does that reveal two modes? –  jerad Dec 14 '12 at 14:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.