0
$\begingroup$

The web-based company I work for has a system that receives packets of information from external sites. I would like to know what approaches can be used to identify patterns or associations between the external sites. For example site A sends a packet on subject 1. Shortly after many other sites may send through packets on the same ID. This may be because Site 2 also received the information from Site 1 and then sent it through to us.

To make it more complicated, it is also possible that once we receive the data we would send it to one or more sites, which may then re-start the process.

The packets contain an ID field that can be used to uniquely identify the subject each packet refers to.

There are hundreds of sites, many of which aggregate packets from each other before sending them on to us. Many of the smaller sites could be excluded from the analysis if that simplifies it. We have a lot of data. Over the course of a few days we may see hundreds of packets all relating to the same ID.

Also, what approaches could be used to best visualise or display the results/patterns to a non-statistical audience? Order may be important. For example if Site 1 always sends through an ID before Site 2, then Site 2 is (possibly) getting its packets from Site 1.

Apologies for the vagueness of the question. I will aim to clarify this question if more details are requested.

Thanks

EDIT: It would also be of interest to look at the analyses for individual sites.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I'm going to approach the part of the question that asks about best visualisation as I have not worked in this area, so cannot recommend approaches, but I have been in audiences at conferences etc where I have been on the receiving end of graphics that are way too complicated.

You will probably need to do some trimming, otherwise you are likely to end up with a graphic that looks like this. This is too complicated an image to present to a non-statistical audience, and I would also venture to suggest that probably any audience is going to get a bit lost as there is just too much detail.

While still a bit complicated, the visualisations towards the bottom of the page here, and here, are much better at conveying an overall picture.

While these don't address order, the size of the nodes can show which sites receive more packets, and the width of the paths can show which links are more important.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. Hmm, perhaps time/order could be visualised by a two-tone chart with the darker colors being the 'leaders' and the lighter colors being the 'followers'. As you said node size could continue to represent volume, and connector width link strength... $\endgroup$
    – Rob Penridge
    Jan 27, 2012 at 22:57
  • $\begingroup$ I'm not sure if it's possible, but maybe you do something with arrowheads, for example at either end of the link. I was thinking that colour might work, but then you'll have quite a few links and the width won't be that great, so using a colour change visualisation is probably not good for this type of graphic. And one can have too much colour as well. :) $\endgroup$
    – Michelle
    Jan 27, 2012 at 23:07
1
$\begingroup$

You might want to take a look at Tauhid R. Zaman's PhD thesis. It's entitled Information Extraction with Network Centralities: Finding Rumor Sources, Measuring Influence, and Learning Community Structure and talks about a number of statistical methods that can be used for networked data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is a great starting point. I'll have to commit some time to go through the whole thing. Good link - thanks. $\endgroup$
    – Rob Penridge
    Jan 27, 2012 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.