All Questions
1,365 questions
12
votes
4
answers
40k
views
Finding outliers without assuming normal distribution
I have small datasets of size 40-50 points. Without assuming that the data is normally distributed I wanted to find out the outliers with 90% confidence at least. I thought boxplot could be a good way ...
- 221
6
votes
2
answers
5k
views
Outlier removal prior to mixed-effect modelling
I'm analysing reaction time data from a grammaticality judgement task (collected in a masked-priming experiment). The stimulus were noun-noun compounds, including 3 types of compounds (depending on ...
- 107
4
votes
2
answers
7k
views
What is a mathematical way to define a point on a scatter plot as an outlier?
I have a graph and there are two points that could be two potential outliers. I'm trying to create a polynomial line of best fit with undefined order.
I believe I could use a >2 standard deviations ...
- 143
2
votes
2
answers
3k
views
Clustering variables with outliers
I am performing a cluster analysis in SAS and some of the variables that I am trying to cluster contain outliers. I've tried to transform the data (log and/or standardize them) but didn't quite work ...
- 578
11
votes
3
answers
2k
views
Good books covering data preprocessing and outlier detection techniques
As the title goes, does anyone know of a good, up to date book that covers data preprocessing in general and especially outlier detection techniques?
The book doesn't need to be focusing exclusively ...
10
votes
3
answers
4k
views
Support vector regression on skewed/high kurtosis data
I'm using support vector regression to model some fairly skewed data (with high kurtosis). I've tried modeling the data directly but I'm getting erroneous predictions I think mainly due to the ...
- 1,941
3
votes
2
answers
987
views
How to identify outliers from a Gumbel distribution with known parameters?
I have a model for data in my experiment that states that the data has a Gumbel distribution with known location and scale.
I am then looking at observations with very high scores that I suspect to be ...
- 1,150
2
votes
0
answers
322
views
Identifying outliers
I have binomial frequency data for an allele associated with populations living in mountainous. These mountains run north to south where sites are nearly fixed for this allele and lowland sites to the ...
- 445
5
votes
3
answers
223
views
Good algorithm for processing positional estimates
We (my team) are building a robot which will navigate around an arena.
The robot uses a camera to determine its position based on markers on the wall. We have tested this and found it can determine ...
- 51
21
votes
3
answers
11k
views
Crash course in robust mean estimation
I have a bunch (around 1000) of estimates and they are all supposed to be estimates of long-run elasticity. A little more than half of these is estimated using method A and the rest using a method B. ...
- 567
4
votes
5
answers
4k
views
Use of robust spread measures such median average deviation and median filters for time series
I have a time series where I need to detect gross anomalies due to coding errors, not small shifts in the structure of the series. I am interested in the most recent data points, not historical data ...
- 61
8
votes
3
answers
5k
views
Outliers spotting in time series analysis, should I pre-process data or not?
My question builds on a previous post on outlier detection in generic time series, and specifically on the answer provided by the always great Rob H.
I work for a small-sized manufacturing company ...
- 729
11
votes
1
answer
3k
views
Automatic feature selection for anomaly detection
What is the best way to automatically select features for anomaly detection?
I normally treat Anomaly Detection as an algorithm where the features are selected by human experts: what matters is the ...
- 3,367
7
votes
4
answers
6k
views
Detect outliers in mixture of Gaussians
I have a ton of univariate samples ($x_i \in \mathbb{R}^+$). I'd like an automated method to check for outliers and identify the outliers, if any are present. A reasonable model for the distribution ...
- 6,738
5
votes
1
answer
17k
views
Handling outliers when comparing two means in a repeated measures design
I am doing a simple study that involved taking a measure at time point 1 and time point 2 (12 weeks later). While the sample was a class, not all members were present at both time points, so I have ...
- 51
7
votes
3
answers
5k
views
Identifying outlier data in high-dimensional settings
I have a data set with high-dimensional feature space. Are there any pre-processing methodologies that can detect outliers from this data set? The outlier, I mean, are the ones that tend to be very ...
- 2,827
26
votes
2
answers
7k
views
Distribution of an observation-level Mahalanobis distance
If I have a multivariate normal i.i.d. sample $X_1, \ldots, X_n \sim N_p(\mu,\Sigma)$, and define $$d_i^2(b,A) = (X_i - b)' A^{-1} (X_i - b)$$ (which is sort of a Mahalanobis distance [squared] from a ...
- 32.3k
6
votes
2
answers
3k
views
Whether to leave the data unaltered in the face of outliers and non-normality when performing structural equation modelling?
I recently received this email from a graduate student, and I get similar questions often enough, that I thought I'd post it here:
I'm using factor analysis, multiple regression, and SEM and ...
- 45.7k
0
votes
2
answers
314
views
Fitting a curve to the edge of a distribution
I need to be able to find outliers in my data. I thought it best to test for this using the Kolmogorov-Smirnov Test.
I have over 800,000 points so I wanted a way to filter the data first to only ...
- 29
3
votes
1
answer
875
views
How to handle outliers in GARCH model?
In GARCH model how should I handle outliers?
Just remove it from my dataset and skip to next data entry?
- 159
3
votes
1
answer
190
views
Find outlier in time domain dataset
We're analyzing a bunch of time domain signals, I want to be able to identify an outlying one.
In our results all the signals will either all be reasonably similar, or in some cases, one should be ...
- 31
4
votes
1
answer
5k
views
How to detect outliers in randomForest regression models?
The randomForest package in R software includes outlier function for the detection of outliers. This function uses proximity matrix or randomForest object for the outlier detection. The manual says ...
- 143
7
votes
1
answer
21k
views
How to deal with outliers?
Background:
There are often some values among sampling set that appear not closely compatible with the rest. They would be called as extreme values or simply outliers. Dealing with outliers has been ...
- 1,444
10
votes
4
answers
688
views
Can one leave out data from research because it is not significant?
I've encountered this sentence while reading an article on sciencemag.org.
In the end, responses from just 7600 researchers in 12 countries were included because the remaining data were not ...
- 3,177
8
votes
2
answers
8k
views
Is it reasonable to delete a large number of outliers from a dataset?
I need some advice on what is a reasonable number of cases to be deleted as outliers.
I have applied outlier detection methods to identify univariate and multivariate outliers from my dataset. ...
- 81
8
votes
3
answers
4k
views
Is it appropriate to identify and remove outliers because they cause problems?
This all pertains to my Psychology honours thesis.
I have two groups (Autism and control) and all participants completed four tasks. It is very important to my study that the groups do not differ on ...
- 81
23
votes
5
answers
15k
views
Whether to delete cases that are flagged as outliers by statistical software when performing multiple regression?
I am performing multiple regression analyses and I am not sure whether outliers in my data should be deleted. The data I am concerned about appear as "circles" on the SPSS boxplots, however there are ...
- 281
3
votes
1
answer
3k
views
Bonferroni for outlier detection?
I am reading a book on time series analysis and I am having problems understanding the section about outlier detection.
The authors say that when you want to know whether at a certain time $T$ there ...
- 11.2k
2
votes
1
answer
468
views
Outlier detection in short time series with two seasonalities
I have short daily time series (less than 4 years) representing sales and exhibiting two seasonalities (weekly and yearly) and I am seeking to identify outliers (not only data reporting errors but ...
- 21
3
votes
1
answer
122
views
How to assess prevalence of grossly inaccurate data in recorded body weights in medical records?
I have around 5 million observations of weights of patients over the course of a year. Some patients have one measurement during that year and others have 30-40 (or more). I've noticed a great deal ...
- 31
0
votes
1
answer
643
views
How to visualize system log data in "phase space"?
First I have to tell my experience in development is extremely thin (a few bash scripts that's all) so bear with me if I can't keep up :)
A little background story to begin:
A few years ago during ...
- 101
5
votes
1
answer
1k
views
How to apply Mahalanobis weighted regression in R?
Some research has shown that in linear regression applications the Mahalanobis distance approach can be used to perform regressions that lower the influence of outliers. The idea is that in the ...
- 3,091
5
votes
1
answer
12k
views
At what value of mean and variance should I throw data away?
I have some score values that are output from a program. There are about 10 such values. The data set is a measure of the "quality" of a speech waveform received over a mobile phone and landline ...
- 295
10
votes
4
answers
872
views
Online outlier detection
I want to process automatically-segmented microscopy images to detect faulty images and/or faulty segmentations, as a part of a high-throughput imaging pipeline. There's a host of parameters that can ...
- 1,977
2
votes
2
answers
262
views
Looking for help identifing outliers in a pilot study to guide future hypothesis testing
I am looking to study a particular type of error on a cognitive test to evaluate for potential clinical implications. As there is no existing research on this variable, I would like to run a pilot ...
- 21
7
votes
2
answers
3k
views
Outlier detection for generic time series
In this case, "generic" being the entire gauntlet of macroeconomic time-series that private and government statistical offices put out.
Some background - I recently started working at a data provider ...
- 2,397
96
votes
6
answers
8k
views
Essential data checking tests
In my job role I often work with other people's datasets; non-experts bring me clinical data and I help them summarise it and perform statistical tests.
The problem I am having is that the datasets I ...
- 5,881
7
votes
2
answers
2k
views
Is there an R package with a pretty function that can deal effectively with outliers?
One of the data sets I deal with is quite strange. The datawarehouse I downloaded the data from has a lot 999999999 values in one of the variables. Apparently the computer system on which the ...
- 726
4
votes
1
answer
1k
views
Tests for consistent measurements and outliers
I have 3 experiments, where some quantity was measured 3 times. Thus, 3 biological replicas, 3 technical replicas in each biological replica, 9 measurements in total. I need to answer the following ...
- 2,634
7
votes
5
answers
7k
views
Computing average value ignoring outliers
This is more of a general statistics question, though if it matters I'm writing PHP code.
Let's say I'm trying to compute the average value of a toy that is commonly bought and sold on the secondary ...
- 173
10
votes
4
answers
5k
views
Does the variable order matter in linear regression [duplicate]
I'm investigating interplay between two variables ($x_1$ and $x_2$). There is a great deal of linear correlation between these variables with $r>0.9$. From the nature of the problem I cannot say ...
- 101
5
votes
1
answer
7k
views
How to get Cook's distance and carry out residual analysis for non-lm() and non-glm() models in R?
I usually use the plot(lm()) or plot(glm()) (combined with par(mfrow=c(2,2)) to analyze ...
- 6,023
8
votes
1
answer
751
views
Weird residuals in linear regression
I analyze a set of multivariate measurements. It is known that several pairs of independent variables show high linear correlation. The graph below shows a scatterplot of one such pair (X and Y, upper ...
- 2,767
15
votes
5
answers
13k
views
Automatic threshold determination for anomaly detection
I am working with a time series of anomaly scores (the background is anomaly detection in computer networks). Every minute, I get an anomaly score $x_t \in [0, 5]$ which tells me how "unexpected" or ...
- 149
8
votes
1
answer
255
views
Is there a name for the high sensitivity of frequency of extreme data points to the mean of a normal distribution?
I remember hearing an argument that if a certain population of people has a mean IQ of 110 rather than the typical 100, it will have far more people of IQ 150 than a similarly-sized group from the ...
- 769
6
votes
2
answers
511
views
Learning a univariate transform (kernel?) for novelty detection
I have 150 observations, 500 features, and I am interested in novelty detection (outlier detection): given a new observation (let's say 'patient') I want to know if it is different from the previous ...
- 875
19
votes
5
answers
2k
views
Can data cleaning worsen the results of statistical analysis?
An increase in the number of cases and deaths occurs during epidemics (sudden increase in numbers) due to a virus circulation (like West Nile Virus in USA in 2002) or decreasing resistance of people ...
- 949
47
votes
8
answers
11k
views
Rigorous definition of an outlier?
People often talk about dealing with outliers in statistics. The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective. For example, ...
- 8,879
7
votes
4
answers
3k
views
Detecting abnormal points in point cloud
I've asked the same question at Math SE, but the suggestion is that probably this question belongs here.
Given a list of point cloud in terms of $(x,y,z)$ how to determine abnormal points?
The ...
- 1,025
8
votes
3
answers
1k
views
How to identify outliers in server uptime performance data?
I have a python script that creates a list of lists of server uptime and performance data, where each sub-list (or 'row') contains a particular cluster's stats. For example, nicely formatted it looks ...
- 165