3
$\begingroup$

I have a data set that includes the number of visits to a website. Here are some descriptive statistics for my data

Median: 4
Mean: 14.1352
SD: 121.8119

Clearly, there are some huge values (individuals who have visited the site thousands of times.) To remove these outliers I considered simply removing any data that falls outside more than 3.5 standard deviations from the mean. The result is that I discovered there is still a significant fat tail with my data. After removal of data that falls outside more than 3.5 standard deviations my descriptive statistics adjust to

Median: 4
Mean: 10.2201
SD: 19.7492

I also explored using a winsorized mean but again since my data is asymmetric I feel like my descriptive statistics are biased. Is there a method that I can use to reevaluate my data to provide descriptive statistics that would represent a ‘majority’ of the population?

As I understand the concept of bootstrapping, I could sample my population and then resample and generation thousands of populations that may represent my population differently based on the resample of my original sample population. Would this method be appropriate?

Any other ideas or direction?

Any references or examples with R would be very much appreciated as well.

$\endgroup$
  • 8
    $\begingroup$ The best thing I think for you to do right now is explain what you're trying to do in general (forget about removing outliers explain what your overall goal is). I highly encourage against removing data unless you know it was somehow recorded incorrectly. $\endgroup$ – Dason Dec 17 '10 at 19:11
  • $\begingroup$ Do you expect your 'bad data' to be from those who visited the site many times or those who viewed it few times. Do you have actual access times? Can you use those to tell the difference between good data and bad data. $\endgroup$ – russellpierce Dec 18 '10 at 19:41
  • $\begingroup$ Same answer as here. $\endgroup$ – user603 Jul 9 '11 at 22:26
5
$\begingroup$

Don't remove any outliers until you explore the data a bit further. I suggest that you should do a log transform on the data and see whether it becomes more nearly symmetrical--the outliers may not be as extreme as you think. (Log values make perfect sense if there is some sort of power law at play.)

$\endgroup$
2
$\begingroup$

The answer you get depends on the question you ask. Dason asked what you are trying to do; I would also ask this question. You say you know some of your data is "tainted" but do you know that this taint applies to the high values? Many sorts of counts have long tails, with no bad data at all.

If you just want to summarize central tendency, you could give mean, median and various trimmed means (the median is just the 50% trimmed mean, after all). Or, perhaps better, you could supply a box plot or dot plot or density plot of the data. Perhaps you do want to take the log; that often makes sense for counts, where you are interested in multiplicative rather than additive differences (e.g. Mary viewed the site 10 times as often as Jim, who viewed it twice as much as Bob)

$\endgroup$
0
$\begingroup$

Dason, I am attempted to develop a clean data set because I do know in fact that some of my data has been tainted. More detail is that some of the visits in my data for a particular ID are from screen scrapers (IP filtering is not working because I do not own the data nor would it be retroactive)

For this reason I am attempting to use a statistical method to scrub out the bad data and create a usable data set. Mostly just looking for clean descriptive statistics of my data.

$\endgroup$
  • 2
    $\begingroup$ This answer should have been made as a comment to Michael's answer or as an edit to your initial question. No big deal - just an FYI. $\endgroup$ – russellpierce Dec 18 '10 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy