4
$\begingroup$

I have a data set that includes the number of visits to a website. Here are some descriptive statistics for my data

Median: 4
Mean: 14.1352
SD: 121.8119

Clearly, there are some huge values (individuals who have visited the site thousands of times.) To remove these outliers I considered simply removing any data that falls outside more than 3.5 standard deviations from the mean. The result is that I discovered there is still a significant fat tail with my data. After removal of data that falls outside more than 3.5 standard deviations my descriptive statistics adjust to

Median: 4
Mean: 10.2201
SD: 19.7492

I also explored using a winsorized mean but again since my data is asymmetric I feel like my descriptive statistics are biased. Is there a method that I can use to reevaluate my data to provide descriptive statistics that would represent a ‘majority’ of the population?

As I understand the concept of bootstrapping, I could sample my population and then resample and generation thousands of populations that may represent my population differently based on the resample of my original sample population. Would this method be appropriate?

Any other ideas or direction?

Any references or examples with R would be very much appreciated as well.

Improve this question
$\endgroup$
3
  • 9
    $\begingroup$ The best thing I think for you to do right now is explain what you're trying to do in general (forget about removing outliers explain what your overall goal is). I highly encourage against removing data unless you know it was somehow recorded incorrectly. $\endgroup$
    – Dason
    Commented Dec 17, 2010 at 19:11
  • $\begingroup$ Do you expect your 'bad data' to be from those who visited the site many times or those who viewed it few times. Do you have actual access times? Can you use those to tell the difference between good data and bad data. $\endgroup$
    – russellpierce
    Commented Dec 18, 2010 at 19:41
  • $\begingroup$ Same answer as here. $\endgroup$
    – user603
    Commented Jul 9, 2011 at 22:26

2 Answers 2

Reset to default
9
$\begingroup$

Don't remove any outliers until you explore the data a bit further. I suggest that you should do a log transform on the data and see whether it becomes more nearly symmetrical--the outliers may not be as extreme as you think. (Log values make perfect sense if there is some sort of power law at play.)

Improve this answer
$\endgroup$
7
$\begingroup$

The answer you get depends on the question you ask. Dason asked what you are trying to do; I would also ask this question. You say you know some of your data is "tainted" but do you know that this taint applies to the high values? Many sorts of counts have long tails, with no bad data at all.

If you just want to summarize central tendency, you could give mean, median and various trimmed means (the median is just the 50% trimmed mean, after all). Or, perhaps better, you could supply a box plot or dot plot or density plot of the data. Perhaps you do want to take the log; that often makes sense for counts, where you are interested in multiplicative rather than additive differences (e.g. Mary viewed the site 10 times as often as Jim, who viewed it twice as much as Bob)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.