All Questions
1,365 questions
3
votes
1
answer
126
views
Estimate the parameters of an ellipse in the presence of large outliers
I have a number of points $x_1,\ldots,x_m\in\mathbb{R}^n$ with weights $w_1,\ldots,w_m$ between 0 and 1. There is an ellipse which contains a very high concentration of points with weights close to ...
- 323
5
votes
1
answer
1k
views
Outlier detection function in R for known distributions
Assuming that I have a one-dimensional data set with a known distribution (i.e. normal, gamma, Weibull, etc.), is there a R function that I can call on the data set that will return the anomalies?
I ...
- 53
2
votes
2
answers
655
views
Removing outliers and calculating a "lowest" attainable price from a pre-determined/fixed time series of prices
Just a foreword, I'm not a mathematician or otherwise statistically skilled. I know my way around calculating standard deviations, but it's all self taught. I'm a programmer with limited stats ...
- 503
3
votes
0
answers
913
views
Outlier detection of an unevenly spaced time series
I found the Rob H answer to this question very interesting and works pretty well. However, I also would like to apply this methodology to an unevenly spaced time series like the following:
...
- 213
0
votes
3
answers
1k
views
Normality of a lognormal variable having a spike in 0?
I have two very right-skewed datasets which I must study for difference in means. Given the skewness, I transformed using log 10 scale after adding 1 to be able to take the log. In other words: ...
- 206
4
votes
2
answers
2k
views
How to identify spikes in a noisy time series?
I have time-series data of brain cell spiking. It's basically got a baseline of random noise with large spikes interspersed. I want to be able to algorithmically cluster the spike portions of the ...
- 241
2
votes
1
answer
972
views
"Convert" Rayleigh random variable into a Uniform random variable?
I have a nested question of sorts. My first question, is that I am wondering if it is possible to 'convert' a Rayleigh random variable into a uniform random variable, and how one may do this.
...
- 1,685
3
votes
0
answers
160
views
Application to a real life problem: identifying outliers in univariate time series data [duplicate]
I have a rather simple problem that I'm having trouble deciding an answer upon. As a student studying Statistics I'm very familiar with terminology and theory but I suppose I'm stumped on my first ...
- 31
10
votes
1
answer
2k
views
How do I incorporate an innovative outlier at observation 48 in my ARIMA model?
I am working on a data set. After using some model identification techniques, I came out with an ARIMA(0,2,1) model.
I used the detectIO function in the package <...
- 147
21
votes
2
answers
11k
views
Anomaly Detection with Dummy Features (and other Discrete/Categorical Features)
tl;dr
What is the recommended way to deal with discrete data when performing anomaly detection?
What is the recommended way to deal with ...
- 313
1
vote
1
answer
1k
views
Removing outliers from discrete data with a lower bound
My data is discrete and has the following distribution:
P(1) = 0.45, P(2) = 0.5, P(3) = 0.02, P(> 3) = 0.02
I want to remove outliers systematically, given the distribution and the fact that 1 is ...
- 941
5
votes
1
answer
8k
views
R: How to interpret the QQplot's outlier numbers?
How to interpret the labels with the outlier numbers when you plot the following in R (QQplot)
...
- 2,399
6
votes
1
answer
228
views
Outliers in importance sampling
I'm working on a HW question in which I'm using the importance sampling method to estimate $E(X)$ where $X$ is distributed as standard Laplace. To do so, I choose my proposal density to be a standard ...
- 2,194
12
votes
2
answers
20k
views
How accurate is IQR for detecting outliers
I'm writing a script that analyses run times of processes. I am not sure of their distribution but I want to know if a process runs "too long". So far I've been using 3 standard deviations of the ...
- 121
5
votes
4
answers
6k
views
Re-check boxplot after outlier removal
I have a sample of 608 subjects and I need to remove outliers for age. In R, the boxplot appears like this:
It shows 74 outliers:
...
- 1,062
104
votes
1
answer
92k
views
Interpreting plot.lm()
I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
- 1,041
4
votes
2
answers
1k
views
Dealing with outliers when comparing variances with Bartlett's test
I have four different groups (with unequal sample sizes of 100 to 120) and want to test if the variance differs.
For ANOVAs I used the winsorized mean to get a more robust estimate and I am wondering ...
- 41
39
votes
1
answer
2k
views
Link Anomaly Detection in Temporal Network
I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
- 705
8
votes
2
answers
3k
views
Median + MAD for skewed data
I am trying to figure out what happens if you apply Hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of Hampel's method over z-...
- 661
24
votes
2
answers
13k
views
Detecting outliers in count data
I have what I naively thought to be a fairly straight forward problem that involves outlier detection for many different sets of count data. Specifically, I want to determine if one or more values in ...
- 241
11
votes
1
answer
9k
views
Histogram with uniform vs non-uniform Bins
This question describes the basic difference between a uniform and a nonuniform histogram. And this question discusses the rule of thumb for picking the number of bins of a uniform histogram that ...
- 233
7
votes
1
answer
4k
views
Checking for outliers in a glmer (lme4 package) with 3 random factors
I have a question relating to the checking for outliers and / or influential points in my dataset using a glmer model with 3 random variables. I'm investigating the ...
- 273
2
votes
1
answer
603
views
Filtering outliers from geo-spatial-temporal log
I have downloaded my Latitude location history from Google for the time of about three years and now I'd like to, for starters, visualize where I've been.
It turns out that the history contains some ...
- 123
6
votes
1
answer
12k
views
How to set SMOTE parameters in R package DMwR?
For different imbalanced data-sets which rare class' proportion differ from 30% (rare) to 5% (rare), what is the best way to define the Perc.Over and ...
- 590
2
votes
1
answer
900
views
Data-driven removal of extreme outliers with Naive Bayes or similar technique
I have a large set of location data from a social network and would like to conduct a mobility study with it. For each object, I have up to several thousand locations where this object posted from. I ...
- 634
1
vote
2
answers
2k
views
Drop a predefined percentage of outliers
I have a dataset that follows something between a power and exponential law. I'm not happy with the IQR method of detection of outliers because on small sets of data (<50-100), it does not give you ...
- 111
4
votes
2
answers
12k
views
How to deal with extreme but "real" data, classify as outliers or no?
I have an explanatory variable, close, which is the daily close price of a firm in the stock market.
The following summarizes this explanatory variable:
...
- 509
1
vote
1
answer
297
views
Select outliers with standard deviation after a nonlinear regression
I've performed an non linear regression with 3 variables and 5 parameters, using Wolfram Mathematica.
Now I want to detect the outliers that are far from 2*(standard deviation) of my function.
I've ...
- 111
5
votes
1
answer
8k
views
Test for bivariate outliers
I have been given a set of data points $(x_i,y_i)$. I have to plot a scatter plot and determine if there are any outliers. But I haven't been taught a method to measure which data point is an outlier ...
- 247
1
vote
1
answer
1k
views
Removal of multi-dimensional outliers?
I have some issues in data reduction, and one expert advised me to remove the outliers and then move to Factor Analysis.
I want to remove outliers together, as I have 61 items, and box plots are not ...
- 11
1
vote
0
answers
112
views
Removing data above the mean
I am working with a data set, for which to remove noisy data I take the average of the sample itself and cut anything above that average. What I'm trying to understand here is does this have a name ...
- 129
1
vote
0
answers
198
views
Asymmetric distribution and PCA [duplicate]
Could someone please explain me this reviewer comment:
asymmetric distribution could affect Principal
Component Analysis results, symmetry of distribution should be
tested. Authors should also ...
- 11
2
votes
2
answers
7k
views
Automatic outlier detection in R
Our model processes millions of multivariate observations; manual outlier detection is impractical. I am looking for a method of automatic outlier detection.
I have been trying to use R package <...
- 145
3
votes
1
answer
2k
views
Skew in both directions and dealing with outliers
I have a lot of wonderfully messy data (got to love the social sciences), and realized that I was not fully prepared to bear its wrath.
For the record, after reading some articles regarding ANOVA, I ...
- 31
3
votes
0
answers
678
views
How should outliers be dealt with in latent growth curve/GMM modeling?
When doing analysis of growth trajectories you often find natural outliers. Often these are not because of any "errors" in the dataset, they are simply there because certain cases grew at an explosive ...
- 3,063
3
votes
1
answer
293
views
What is the explanation for a regressor losing statistical significance when a high leverage point is dropped?
I'm currently working on an Econometrics project and I've come to a point where I've dropped a high leverage point as identified by cook's distance and a leverage plot (had observations that were ...
- 51
0
votes
2
answers
257
views
Is this outlier merely unusual or does it signify manipulation?
These are actual reported costs (60) for a series of projects for two years, say USD (though actually not) totalling $57,975,403.24:
...
- 177
4
votes
1
answer
1k
views
Are there terms to distinguish between the two types of outliers?
I hate talking about "outliers," because I view that term as encompassing two entirely different concepts. The first is when it refers to data that was incorrectly recorded or measured. ...
- 4,791
1
vote
0
answers
898
views
Fitting a surface to 3D data
I am building a program to predict the score in a game of twenty20 cricket. I have a series of 3D datapoints, calculated from a number of games I have data for.
Along the x axis we have number of ...
- 243
7
votes
6
answers
21k
views
How to detect outliers in skewed data set?
I am working on my school datamining project. Within preprocessing stage I need to remove outliers from my data set which is positively skewed (see description). I have an idea to remove all values ...
- 269
1
vote
0
answers
378
views
Hypothesis testing to determine cluster outliers
I have a cluster of $p$-dimensional data from $n$ samples which is assumed to be normally distributed as a multivariate Gaussian with sample mean ${\bar{\mu}}$ and sample covariance matrix ${\bar{S}}$....
- 1,249
6
votes
1
answer
1k
views
Robust parameter estimation for Exponentially modified Gaussian distribution
I'd like to test how well my data can be modeled by an Exponentially modified Gaussian distribution (Wikipedia) or Normal-exponential-gamma (NEG) Distribution. However, the parameter estimation (which ...
- 3,398
17
votes
3
answers
7k
views
Estimating parameters of a normal distribution: median instead of mean?
The common approach for estimating the parameters of a normal distribution is to use the mean and the sample standard deviation / variance.
However, if there are some outliers, the median and the ...
- 3,398
7
votes
3
answers
7k
views
How to estimate the parameters of a Gaussian distribution sample with outliers?
I have a sample of length $N$, mostly taken from a Gaussian distribution with unknown mean and variance. Of those $N$ samples, some proportion of them (typically less than 1-2%) are outliers, taken ...
- 171
11
votes
2
answers
23k
views
Influential residual vs. outlier
First, I should state that I have searched on this site for the answer. I either didn't find a question that answered my question or my knowledge level is so low I didn't realize I already read the ...
- 559
9
votes
1
answer
2k
views
Does Pearson correlation require removal of bivariate or univariate outliers?
Does the Pearson's correlation estimator require no bivariate outliers, or no outliers in each of two individual vectors of data? The answer will impact on how I winsorize outliers before calculating ...
- 2,286
55
votes
4
answers
14k
views
Fast linear regression robust to outliers
I am dealing with linear data with outliers, some of which are at more the 5 standard deviations away from the estimated regression line. I'm looking for a linear regression technique that reduces the ...
- 3,284
4
votes
1
answer
11k
views
Handling outliers in ANOVA
I have a question relative to the correct method to deal with univariate outliers when one has to conduct an ANOVA.
Starting with an example, suppose I have two samples of subjects tested on a number ...
- 1,062
4
votes
1
answer
38k
views
Best way to display data with outliers?
I am fairly new to data analysis and visualization, and I'm trying to figure out the best model to show some data regarding page load time (in seconds).
The current view is a line graph where the x-...
- 145
0
votes
1
answer
1k
views
How to find outliers in a data series?
I have a series of 100 points
My dataset can be found here . Each row is a data series. The plot for 90th row is
It's easy to detect outliers visually by plotting example. I tried using hampel ...
- 23