1
$\begingroup$

Short version

I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc.

Long version

Currently I am facing data which according to A Brief Survey on Sequence Classification by Xing et. al. can be called "complex event sequences".

The key points:

  • The data basically describes an interaction between users and websites (user actions), like visiting certain pages, clicking on links etc.
  • For every user action, a set of different properties is available (like e.g. main text on the website, category etc.)
  • Since it is a website, there is no definitive path to go. In order to show interest e.g. for the "news" section, they can browse the news-articles in any order.
  • The data does not cover all user interactions with the site, only specific ones.

I want to understand how users behave or what does lead to specific goals (e.g. subscribing for a newsletter).

If the data would be tabular, I would (using RStudio atm)

  • browse around
  • build white box classification models (if a specific goal is involved)
  • look at summary statistics
  • look at various kind of simple plots
  • forming hypotheses along the way
    • look for cues supporting or rejecting these
    • ask for feedback from domain experts
  • After I got a selection of promising hypotheses, I would test these on the website by changing content / appearance (via so called A/B-Tests)

(This is what I like to call explorative data analysis, not exactly sure if this is correct)

Now ...

The complex sequence structure of the data makes it very hard to perform the exploration part in the same manner. One approach (I have used so far) is to

  1. Convert the sequence data into a table by creating a specific set of features
  2. Perform the described explorative data analysis
  3. If(not satisfied) GOTO 1

This approach is cumbersome. Even when I find a way to convert the multiple properties of user actions into features, I get stuck when I want to explore truly sequential patterns like

"VISIT entertainment pages AFTER visiting medicine pages"

or

"Users which return to the same website more than once in 1 week are most likely to subscribe for newsletter"

The maddening part is, that for every possible pattern which comes to my mind, I can create a feature (by writing code). But I have got the feeling that this is not exploration. Instead of roaming through the jungle, I use a helicopter to hop to places previously pointed out on a map.

I admit I am overwhelmed by the data complexity. So now I am looking for anything which which may help to get started.

  • (Standard) Procedures
  • Feature engineering tactics
  • Tools
  • Books
  • Explorative Sequence Analysis War Stories
  • Keywords leading to more papers etc.

The domain does not matter

TL;DR;

I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc.

$\endgroup$

1 Answer 1

2
$\begingroup$

First of all: stop and breathe. I agree with "there is no too much data" but I also agree that a lot of data can drive us mad.

I think the your first resources should be a pen and a paper. For many people this may sound ridiculous but in my experience it really works. Start for putting in writing some of the questions you would like to be answered, example: 1 - Users that return to the website 2 - Users that subscribe to a newsletter 3 - Users that visit the news section 4 - .... This exercise will help you to create a structure in the data. If you don't have a data model then you can create one. They are very useful. Go step by step, if latter on you decide that you need more variables (that I'm sure you will) than you just create them. You don't have to have everything at first.

For starting you can also browse around, there is nothing wrong with that. Maybe that makes you feel more conformable with the data and then you start having more ideas.

As material, try a Data Mining book. Check these ones "Data Mining: Concepts and Techniques - Jiawei Han" and "HANDBOOK OF STATISTICAL ANALYSIS AND DATA MINING APPLICATIONS - ROBERT NISBET, JOHN ELDER, GARY MINER". They are quite complete and maybe is a starting point.

$\endgroup$
4
  • $\begingroup$ Also users don't rationally decided that they will click on a specific part of a page. It is all subconscious. $\endgroup$ Jun 18, 2015 at 16:55
  • $\begingroup$ @yohanna Thank you for reply. I already use your described approach, but I got the feeling that people smarter than me have figured out how to do in a better way (using special tools ?). $\endgroup$
    – mlwida
    Jun 19, 2015 at 7:02
  • $\begingroup$ @missingdataguy Yes, it is not deterministic, but probabilistic. But in all subconsciousness, patterns do emerge (if one likes on article on a particular topic, he may like a similar one). $\endgroup$
    – mlwida
    Jun 19, 2015 at 7:04
  • $\begingroup$ @steffen Data analysis is subjective and is a process. I believe that you should always start with the basics and at your own time you will get more "sophisticated". At every step of the way you will feel the need to refine your methods and you will start using other areas for data analysis. One area that it might be interesting for you is pattern recognition. Have you thought about that? And there is no such thing as "smarter" people, believe me, just more experienced people. $\endgroup$
    – yohanna
    Jun 19, 2015 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.