All Questions
1,365 questions
4
votes
2
answers
3k
views
Removing outliers from asymmetric data
I have a data set that includes the number of visits to a website. Here are some descriptive statistics for my data
Median: 4
Mean: 14.1352
SD: 121.8119
Clearly, there are some huge values (...
Brandovious
8
votes
2
answers
2k
views
Method to reliably determine abnormal statistical values
I'm searching for a statistical method to determine if a player is cheating in an online game. The game is a Quake3 like game (ego-shooter).
Given a number of positive points and a number of negative ...
- 183
11
votes
3
answers
4k
views
Finding the average GPS point
I need to write a program to find the average GPS point from a population of points.
In practice the following happens:
Each month a person records a GPS point of the same static asset.
Because of ...
- 213
8
votes
3
answers
396
views
Dealing with "trouble maker" samples
I have a pretty large data set (~300 cases with ~40 continuous attributes, binary labeled) which I used to create several alternative predictive models. To do this, the set was divided to training and ...
- 2,767
9
votes
1
answer
11k
views
On univariate outlier tests (or: Dixon Q versus Grubbs)
In (most of) the analytical chemistry literature, the standard test for detecting outliers in univariate data (e.g. a sequence of measurements of some parameter) is Dixon's Q test. Invariably, all the ...
19
votes
3
answers
10k
views
Robust outlier detection in financial timeseries
I'm looking for some robust techniques to remove outliers and errors (whatever the cause) from financial time-series data (i.e. tickdata).
Tick-by-tick financial time-series data is very messy. It ...
- 371
89
votes
14
answers
7k
views
Why haven't robust (and resistant) statistics replaced classical techniques?
When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so ...
- 10.7k
109
votes
13
answers
86k
views
Simple algorithm for online outlier detection of a generic time series
I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren'...
- 2,001
30
votes
4
answers
13k
views
Application of wavelets to time-series-based anomaly detection algorithms
I've been beginning to work my way through Statistical Data Mining Tutorials by Andrew Moore (highly recommended for anyone else first venturing into this field). I started by reading this extremely ...
- 949
16
votes
4
answers
5k
views
Separating two populations from the sample
I'm trying to separate two groups of values from a single data set. I can assume that one of the populations is normally distributed and is at least half the size of the sample. The values of the ...
- 359
36
votes
4
answers
11k
views
Why isn't RANSAC most widely used in statistics?
Coming from the field of computer vision, I've often used the RANSAC (Random Sample Consensus) method for fitting models to data with lots of outliers.
However, I've never seen it used by ...
- 687
104
votes
13
answers
73k
views
What is the best way to identify outliers in multivariate data?
Suppose I have a large set of multivariate data with at least three variables. How can I find the outliers? Pairwise scatterplots won't work as it is possible for an outlier to exist in 3 dimensions ...
- 58.3k
99
votes
9
answers
235k
views
How should outliers be dealt with in linear regression analysis?
Often times a statistical analyst is handed a set dataset and asked to fit a model using a technique such as linear regression. Very frequently the dataset is accompanied with a disclaimer similar to ...
- 4,454
18
votes
6
answers
11k
views
Correcting for outliers in a running average
We have a daemon that reads in data from some sensors, and among the things it calculates (besides simply just reporting the state) is the average time it takes for the sensors to change from one ...
Edward Z. Yang
100
votes
13
answers
166k
views
What do you call an average that does not include outliers?
What do you call an average that does not include outliers?
For example if you have a set:
{90,89,92,91,5} avg = 73.4
but excluding the outlier (5) we have
<...
- 1,103