I'm a journalist turned developer who hobbies in APIs and analysis of web traffic. I've always enjoyed learning about stats but as I learned, I learned that I have misapplied some basic concepts in the past. I now know a bit better, and know to doublecheck my ideas with those are smarter than me -- I'm hoping to find a great answer to whether my idea is going to work.
A while back I was using traffic data to do basic summary statistics and standard deviation as a basic measure of variability. Where I went wrong was that I tried to create a confidence interval -- wrong because I presumed my dataset was Gaussian when in fact I now believe it to resemble something more like a Power Law. So I could sample the data and create standard deviations of the data all day but and they wouldn't be good models.
Recently I've been thinking about this problem. Given a limited set of data, how can I try to model the traffic for a website in question? I have built an API on top of a wonderful dataset, and that dataset is used in all kinds of pages around an actively used site. My idea: use the API access pattern as a small sample of my larger data set.
If you're following, what I'd do is basically treat each datarequest as a sample, maybe grouping requests by hour and logging total, standard deviation, number of samples n, etc. Then take use standard deviation of those samples to create the model, under the assumption that the Central Limit Theorem says that distribution will be normal.
As I understand it, even though my dataset is not normal the statistics I derive from that dataset should be. Is that the case? If so, can I create a confidence interval from that data?