An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.
0
votes
2answers
44 views
Re-check boxplot after outlier removal
I have a sample of 608 subjects and I need to remove outliers for age. In R, the boxplot appears like this:
It shows 74 outliers:
...
3
votes
1answer
38 views
Dealing with outliers when comparing variances with Bartlett's test
I have four different groups (with unequal sample sizes of 100 to 120) and want to test if the variance differs.
For ANOVAs I used the winsorized mean to get a more robust estimate and I am wondering ...
3
votes
0answers
37 views
Link Anomaly Detection in Temporal Network
I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
4
votes
2answers
63 views
Median + MAD for skewed data
I am trying to figure out what happens if you apply hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of hampel's method over ...
0
votes
0answers
98 views
High kurtosis, skewness and outliers
Currently I am working on my master this which is about excess returns (Sharpe ratio) of Asian REITs. I just transformed all the data in variables which are ready to use in SPSS. In the panel data ...
11
votes
1answer
226 views
Detecting Outliers in Count Data
I have what I naively thought to be a fairly straight forward problem that involves outlier detection for many different sets of count data. Specifically, I want to determine if one or more values in ...
4
votes
1answer
95 views
Histogram with uniform vs non-uniform Bins
This question describes the basic difference between a uniform and a nonuniform histogram. And this question discusses the rule of thumb for picking the number of bins of a uniform histogram that ...
3
votes
1answer
108 views
Checking for outliers in a glmer (lme4 package) with 3 random factors
I have a question relating to the checking for outliers and / or influential points in my dataset using a glmer model with 3 random variables. I'm investigating the ...
0
votes
0answers
10 views
Checking for outliers in a glmer (lme4 package) with 3 random factors [duplicate]
I have a question relating to the checking for outliers and/or influential points in my dataset using a glmer model with 3 random variables. I'm investigating the detection rate (SumDetections) of ...
2
votes
1answer
39 views
Filtering outliers from geo-spatial-temporal log
I have downloaded my Latitude location history from Google for the time of about three years and now I'd like to, for starters, visualize where I've been.
It turns out that the history contains some ...
0
votes
0answers
56 views
How to set SMOTE parameters in R package DMwR?
For different imbalanced data-sets which rare class' proportion differ from 30% (rare) to 5% (rare), what is the best way to define the Perc.Over and ...
1
vote
1answer
75 views
Data-driven removal of extreme outliers with Naive Bayes or similar technique
I have a large set of location data from a social network and would like to conduct a mobility study with it. For each object, I have up to several thousand locations where this object posted from. I ...
1
vote
1answer
58 views
Drop a predefined percentage of outliers
I have a dataset that follows something between a power and exponential law. I'm not happy with the IQR method of detection of outliers because on small sets of data (<50-100), it does not give you ...
1
vote
2answers
159 views
How to deal with extreme but “real” data, classify as outliers or no?
I have an explanatory variable, close, which is the daily close price of a firm in the stock market.
The following summarizes this explanatory variable:
...
1
vote
1answer
56 views
Select outliers with standard deviation after a nonlinear regression
I've performed an non linear regression with 3 variables and 5 parameters, using Wolfram Mathematica.
Now I want to detect the outliers that are far from 2*(standard deviation) of my function.
I've ...
3
votes
1answer
119 views
Test for bivariate outliers
I have been given a set of data points $(x_i,y_i)$. I have to plot a scatter plot and determine if there are any outliers. But I haven't been taught a method to measure which data point is an outlier ...
1
vote
1answer
62 views
Multiple dimension outliers removal?
I have some issues in data reduction, and one expert advised me to remove the outliers and then move to Factor Analysis. I want to remove outliers together, as I have 61 items, and box plots are not ...
0
votes
0answers
38 views
Removing data above the mean
I am working with a data set, for which to remove noisy data I take the average of the sample itself and cut anything above that average. What I'm trying to understand here is does this have a name ...
0
votes
3answers
337 views
Automatic outlier detection in R
Our model processes millions of multivariate observations; manual outlier detection is impractical. I am looking for a method of automatic outlier detection.
I have been trying to use R package ...
2
votes
1answer
179 views
Skew in both directions and dealing with outliers
I have a lot of wonderfully messy data (got to love the social sciences), and realized that I was not fully prepared to bear its wrath.
For the record, after reading some articles regarding ANOVA, I ...
2
votes
0answers
39 views
How should outliers be dealt with in latent growth curve/GMM modeling?
When doing analysis of growth trajectories you often find natural outliers. Often these are not because of any "errors" in the dataset, they are simply there because certain cases grew at an explosive ...
1
vote
1answer
76 views
What is the explanation for a regressor losing statistical significance when a high leverage point is dropped?
I'm currently working on an Econometrics project and I've come to a point where I've dropped a high leverage point as identified by cook's distance and a leverage plot (had observations that were ...
1
vote
2answers
149 views
Is this outlier merely unusual or does it signify manipulation?
These are actual reported costs (60) for a series of projects for two years, say USD (though actually not) totalling $57,975,403.24:
...
3
votes
0answers
72 views
Are there terms to distinguish between the two types of outliers?
I hate talking about "outliers," because I view that term as encompassing two entirely different concepts. The first is when it refers to data that was incorrectly recorded or measured. For ...
1
vote
0answers
60 views
Fitting a surface to 3D data
I am building a program to predict the score in a game of twenty20 cricket. I have a series of 3D datapoints, calculated from a number of games I have data for.
Along the x axis we have number of ...
1
vote
6answers
539 views
How to detect outliers in skewed data set?
I am working on my school datamining project. Within preprocessing stage I need to remove outliers from my data set which is positively skewed (see description). I have an idea to remove all values ...
1
vote
0answers
84 views
Hypothesis testing to determine cluster outliers
I have a cluster of $p$-dimensional data from $n$ samples which is assumed to be normally distributed as a multivariate Gaussian with sample mean ${\bar{\mu}}$ and sample covariance matrix ...
3
votes
0answers
124 views
Robust parameter estimation for Exponentially modified Gaussian distribution
I'd like to test how well my data can be modeled by an Exponentially modified Gaussian distribution (Wikipedia) or Normal-exponential-gamma (NEG) Distribution. However, the parameter estimation (which ...
7
votes
2answers
344 views
Estimating parameters of a normal distribution: median instead of mean?
The common approach for estimating the parameters of a normal distribution is to use the mean and the sample standard deviation / variance.
However, if there are some outliers, the median and the ...
0
votes
0answers
31 views
Anomaly prediction confidence for frequentist vs bayesian parameter inference
I am comparing the behavior of some implementations of Bayesian and frequentist approaches to parametric anomaly detection and currently trying to figure out the differences when the sample set is ...
3
votes
3answers
237 views
How to estimate the parameters of a Gaussian distribution sample with outliers?
I have a sample of length $N$, mostly taken from a Gaussian distribution with unknown mean and variance. Of those $N$ samples, some proportion of them (typically less than 1-2%) are outliers, taken ...
6
votes
2answers
317 views
Influential residual vs. outlier
First, I should state that I have searched on this site for the answer. I either didn't find a question that answered my question or my knowledge level is so low I didn't realize I already read the ...
5
votes
1answer
207 views
Does Pearson correlation require removal of bivariate or univariate outliers?
Does the Pearson's correlation estimator require no bivariate outliers, or no outliers in each of two individual vectors of data? The answer will impact on how I winsorize outliers before calculating ...
16
votes
3answers
502 views
Fast linear regression robust to outliers
I am dealing with linear data with outliers, some of which are at more the 5 standard deviations away from the estimated regression line. I'm looking for a linear regression technique that reduces the ...
0
votes
0answers
31 views
Reasons for fitting results (anti image)
I need to interpret a residual table which was fitted by an analyst plotting $y_.0123$ (final residual) against the $x_{1.023}$. The table represents residuals produced by each variable.
...
2
votes
1answer
330 views
Handling outliers in ANOVA
I have a question relative to the correct method to deal with univariate outliers when one has to conduct an ANOVA.
Starting with an example, suppose I have two samples of subjects tested on a number ...
3
votes
1answer
256 views
Best way to display data with outliers?
I am fairly new to data analysis and visualization, and I'm trying to figure out the best model to show some data regarding page load time (in seconds).
The current view is a line graph where the ...
0
votes
1answer
212 views
How to find outliers in a data series?
I have a series of 100 points
My dataset can be found here . Each row is a data series. The plot for 90th row is
It's easy to detect outliers visually by plotting example. I tried using hampel ...
0
votes
3answers
78 views
How to use LOF for outlier detection as I have training and test dataset?
I want to use the Local Outlier Factor (LOF) algorithm for outlier detection but it simply finds outliers on unlabed data as whole and you do not need to have a training and test set. However in my ...
0
votes
0answers
98 views
Any out source outlier (anomaly) detection package for Weka?
There is KDD'98 cup data waiting for me to run some anomaly detection algorithms but Weka does not include any of them, natively. Is there any 3rd part package that can be integrated to weka?
1
vote
2answers
161 views
Which does a datamining package support anomaly detection?
I aim to have some anomaly detection process on my data but Weka, Rapidminer or Knime do not support anomaly detection algorithms. How would I take care of the process?
2
votes
1answer
195 views
How to test for outliers in an mlogit model in R
I am running a multinomial logistic regression using the mlogit package and mlogit function in R. Now I need to check for ...
3
votes
1answer
146 views
Robust outlier detection in curve fitting with correlated errors
Assume I have data originating from a model
$$
y_i=f(t_i)+e(t_i)
$$
with $f\in C_2(\mathbb{R})$.
The only thing I know about the errors is that they roughly happen to be of to different sources:
...
1
vote
0answers
130 views
Unsupervised anomaly detection with factor analysis (in R)
The basic idea i'm trying is to model the data with factor analysis, assuming a latent variable structure that underlies the observations. Labels for "real" anomalies are available and used for ...
1
vote
0answers
265 views
Standardising the removal of outliers from a small data set
I have been wading through the many discussions on outliers on this site but I am still unfortunately having difficulty determining what to do with my data set.
My study consists of a simple pre-post ...
1
vote
1answer
93 views
Dataset and outlier question
I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule
targets datasets of continuous (not necessarily ...
6
votes
1answer
171 views
Identifying fraudulent questionnaires
Questionaires are often used in social sciences.
Many people try to complete them very quickly and very often they only "guess" answers.
Is there any statistical technique or any research in this ...
1
vote
1answer
46 views
If I have good reason to expect a certain value as the sample mean and I obtain something very different, should I obtain another sample?
In my specific case, I am measuring the mean of a sample for different values of a factor (let's say the possibilities are integers from 1 to 100). I have very good reason to expect the mean of each ...
6
votes
3answers
198 views
How to add outliers to an existing data?
I want to test few similarity measures for outlier detection. I've got some data from UCI repository, for example: Breast-Cancer.
Is there a smart way to add artificial outliers to an existing data?
...
3
votes
2answers
193 views
Detecting initial trend or outliers
In my test procedure I sequentially take 10 measurements of a recently perturbed physical system, and I often find the first few (between 0 and 4) measurements can be inaccurate because the system has ...
