Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite

I'm a journalist turned developer who hobbies in APIs and analysis of web traffic. I've always enjoyed learning about stats but as I learned, I learned that I have misapplied some basic concepts in the past. I now know a bit better, and know to doublecheck my ideas with those are smarter than me -- I'm hoping to find a great answer to whether my idea is going to work.

A while back I was using traffic data to do basic summary statistics and standard deviation as a basic measure of variability. Where I went wrong was that I tried to create a confidence interval -- wrong because I presumed my dataset was Gaussian when in fact I now believe it to resemble something more like a Power Law. So I could sample the data and create standard deviations of the data all day but and they wouldn't be good models.

Recently I've been thinking about this problem. Given a limited set of data, how can I try to model the traffic for a website in question? I have built an API on top of a wonderful dataset, and that dataset is used in all kinds of pages around an actively used site. My idea: use the API access pattern as a small sample of my larger data set.

If you're following, what I'd do is basically treat each datarequest as a sample, maybe grouping requests by hour and logging total, standard deviation, number of samples n, etc. Then take use standard deviation of those samples to create the model, under the assumption that the Central Limit Theorem says that distribution will be normal.

As I understand it, even though my dataset is not normal the statistics I derive from that dataset should be. Is that the case? If so, can I create a confidence interval from that data?

share|improve this question
3  
Why not use robust, non-parametric summaries like median, IQR, MAD, letter statistics, and so on? They are easy to compute, easy to understand, are defined for all distributions (power laws might not even have SDs), resist the effects of outliers, and often are more efficient at estimating distributions when parametric assumptions (like Gaussianity) break down. – whuber♦ May 6 '11 at 17:19
I'll have to read up before I can properly answer that question, but I am looking at confidence intervals because I would like to create a running model of what is normal and what kind of traffic is significantly interesting. I feel like a simple expected range like "10,000 to 20,000 pageviews" would be really easy to digest. – editor May 6 '11 at 17:34
3  
A confidence interval would not even answer the question of what trafic is "normal". You need a prediction interval for that. And the use of percentiles is the most robust way of doing that, especially if you have loads of data. – Aniko May 6 '11 at 17:55

1 Answer

up vote 1 down vote

There are a few variants on the Central limit theorem, however an important point is whether or not you have a independent and identically distributed sample. Referrer engines and SEO seems to always be changing the sample, even though your theoretical target population may remain the same. You cannot make this assumption when sampling from a web site blindly. There is some flexibility to the CLT, but at some point it will break. You might be better off studying your population first, and hopefully you will be able to explaining who they are or where they come from.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.