Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

412 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
33
votes
1answer
1k views

Link Anomaly Detection in Temporal Network

I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
7
votes
0answers
796 views

Detecting outliers using 95% PI around a natural spline fit

Please see the picture below: I wanted to mark the points that are not consistent with their adjacent points as outlier. What I did was to fit a natural spline fit to 1000 observations (the purple ...
5
votes
1answer
1k views

Unsupervised outlier detection in 2D space

Problem I'm working on a school project in Java and my goal is to detect and remove outliers from a dataset containin geo points. The final result should be a single cluster, with any shape, ...
5
votes
0answers
2k views

Detecting outliers in non-normal distribution data

I'm working with data from a resistivity test. However, during the test it is common that a few measurement points are wrong due to technical failure. So I want to find and remove these points. I ...
4
votes
0answers
75 views

Is this method for comparing whether two distributions have similar outliers studied in the literature?

I am working on a project where I am trying to compare outliers from two different distributions. I came up with a natural seeming measurement, and I want to find out whether there's a name for it or ...
4
votes
0answers
108 views

“Forward search” methods for outlier detection etc. in regression

I am reading Robust Diagnostic Regression Analysis by Anthony Atkinson and Marco Riani. They propose a robust "forward search" method for detecting outliers and other problematic data in regression (...
4
votes
0answers
83 views

Can I statistically describe a single case/outlier vs. a distribution?

I have a dataset consisting of body weight and corresponding age for a bunch of healthy subjects (grey triangles below). I fit a nonlinear function to this data and graphed a 95% prediction interval. ...
4
votes
0answers
2k views

Influence plot for potential outlier detection from logistic regression in R

I am looking into identifying extreme values from their contribution to a binary outcome model. I have an unbalanced set and some extreme values which are part of the smaller set to predict (i.e ...
4
votes
0answers
80 views

Outlier removal using a notion of statistical distance?

Suppose one knows that a distribution is essentially a multivariate Gaussian, but that some of the points are contaminated. The theory of M-estimation shows that if you start with a possibly crude ...
4
votes
1answer
476 views

Changing sensitivity (cval) in tsoutliers resulting in unexpected results

I am using the excellent tsoutliers R package to detect outliers (additive outliers, temporary changes etc.), but the cval ...
4
votes
0answers
5k views

When the Median Absolute Deviation (MAD) is zero

Suppose my data look like the following: (10, 10, 10, 10, 10, 0) Would it be possible to remove an outlier in this distribution using the median absolute deviation? Of course, you wouldn't need to ...
4
votes
0answers
263 views

what to do with ridiculous but valid leverage points

So I'm having some difficulty fitting a linear model to the data (see other post here glm model fit - can't find a family/link combination that produces good fit). In particular, I'm worried ...
4
votes
0answers
444 views

Basic methods for detecting outliers

Let $X$ be a matrix of $n$ rows and $m$ columns. $n$ is the number of samples and $m$ is the number of gene expressions. Gene expressions are basically numerical continuous values. Assume we have a ...
4
votes
0answers
16k views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
4
votes
0answers
301 views

Can I use weights generated by robust regression in a quasipoisson glm in R?

I have response variable count data that should be treated as quasipoisson or something similar. This data also contains outliers which are important to the dataset. I cannot find an r package that ...
4
votes
0answers
2k views

How to assess the statistical significance of a single data point

I am not very well knowledgeable in statistics as I have yet to take a formal class in it (but have signed up for one next year) and yet find myself in need of finding out whether or not a single data ...
4
votes
1answer
105 views

Remove Outlying Data with a Different Trend

I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one ...
4
votes
1answer
473 views

Semi-supervised method for identifying states and state durations in a time series for anomaly detection

I am developing a semi-supervised method for identifying anomalies in a time series with multiple states. Let's consider this example time series in which there are two states e.g. state 1 and 2 with ...
4
votes
1answer
1k views

Outliers and Influential observations in fixed effects regression

I am running a fixed effects regression with a very unbalanced panel data. There are a lot of large residuals. For half of my observations, the residuals are large. However, I do not want to simply ...
3
votes
1answer
29 views

Outlier detection in point estimates

I have to perform outlier detection on population estimates for certain variables at the city level. For example, I might be estimating median income for a city and I want to know if there are any ...
3
votes
0answers
34 views

Estimating tail deviation from Q-Q plots

I am running experiments and for certain cases am able to find a suitable distribution for the data. However, in most cases, depending on a certain parameter, the observed vs fitted distributions have ...
3
votes
0answers
42 views

Appropriate approach for identifying outliers in health board performances with COVID-19 mortality rates

I am relatively new to statistics but am very keen to learn the best approaches with working examples as I prefer to learn this way. We are clearly living in an unprecedented time with COVID-19 and I ...
3
votes
0answers
61 views

Resources for learning the time series stuff they don’t (or didn’t) teach you

I at one point, a long time ago, had two years of graduate econometrics focusing on time series, plus more on micro cross-section techniques. I haven’t made much use of the time-series stuff for a ...
3
votes
0answers
33 views

Using robust regression to detect outliers

Rousseeuw and van Zomeren (1990) propose using robust regression to detect multivariate outliers, particularly in OLS regression. This approach seems to make sense (although I have not studied it in ...
3
votes
0answers
1k views

What are the different influences of outliers regarding the feature scaling methods: standardization VS. normalization?

I've come to know that normalization (MinMax scaling) and standardization (Z-score normalization) on data have different influences from outliers in the data. In About Feature Scaling and ...
3
votes
0answers
2k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
3
votes
0answers
484 views

Difference between identifying outliers using LOF and K-means clustering

I am identifying outliers using K-means and LOF (Local Outlier Factor). Let's say if we are identifying possible outliers using both the techniques, I believe LOF will pick global outliers also as ...
3
votes
0answers
942 views

Treating outliers for time series forecasting in Python

What is the best way to treat outliers in a time series forecasting model? In particular, for product demand modeling? Based on what I've read so far, the following methods can be applied: ...
3
votes
0answers
1k views

Algorithm to detect time series anomalies (outliers) (using Apache Spark)

I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and ...
3
votes
1answer
1k views

Polynomial fit: removing outliers

I want to fit a scatter plot with a polynomial, and find the correlation between two variables. 1) How can I define and remove outliers from data points? (in the figure the outliers on the right ...
3
votes
0answers
49 views

Influential observation (strategy) in linear regression model

Suppose I have dataset(518 observations): ...
3
votes
0answers
176 views

May outliers represent significant values?

Each element of a 56x1 vector represents the functional association between two brain networks. I want to assess which of these 56 values are significant. One way to deal with this is to use an ...
3
votes
0answers
802 views

Bonferroni adjustment of studentized residuals for outlier detection

Here is the statement that I have read (source pp. 21-23): Since we are selecting the furthest outlier, it is not legitimate to use a simple t-test(for studentized residuals) for detecting ...
3
votes
0answers
127 views

Include or exclude residuals that are zero

Objective: Working with time series of historical price data that is liable to have outliers and wish to apply a defined procedure to identify the outliers. Procedure: Take time series and apply ...
3
votes
0answers
2k views

Detecting outliers in percentages

My dataset looks like below - Total Success Percentage 100 65 65% 50 25 50% 30 20 66.6% 50 40 80% Plot - Each row is ...
3
votes
0answers
95 views

Are factorization machines robust to outliers?

Factorization machines (FMs) seem great for modeling very sparse data. However, I have not come across much discussion regarding the impact of outliers. If FMs are robust, why is that so?
3
votes
0answers
873 views

Identifying outliers in logistic regression model

I'm looking to identify outliers in a logistic regression model, e.g. ...
3
votes
0answers
233 views

Detection of noise and outliers

I am measuring the number of cells with a mutation in a series of 106 subjects. For each position of the genome, the method will output the total number of cells analysed and the number of cells with ...
3
votes
0answers
712 views

R package for classification and outlier detection together

I have a similar problem as this one. My training samples contain N observations and K>2 classes. I want to classify my test samples into one of the K classes, or as an outlier if it is far from any ...
3
votes
0answers
124 views

How should I interpret/follow-up on mixed logistic regression (GLMM) diagnostics?

I have experimental data (n subjects = 64) in which the response variable, accuracy (0 or 1), was measured 9 times within subjects. My predictor is Condition (A vs B) measured between subjects. I ...
3
votes
0answers
209 views

Rules of thumb for a proportion of outliers depending on the dimension

I am implementing and benchmarking different "robust" PCA (principal component analysis, see for instance Robust Principal Component Analysis?) for data that should align (I have no prior on the ...
3
votes
0answers
5k views

Unbalanced Panel data using R - Removing outliers and heteroskedastcity

I am new in R and it’s my first time using it so I’ll appreciate the help. I am estimating income elasticity for electricity consumption using budget shares. I have data for 8 regions categorized into ...
3
votes
0answers
63 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
3
votes
0answers
988 views

Multivariate outlier detection for PLS model

I am working with a PLS model (library pls) in R, where I am developing calibration models for NIRS data. I have been using other commercial software before that allowed me to detect outliers based on ...
3
votes
1answer
180 views

Von Mises distribution to detect outliers

I am working out the difference between two angles from a circle, and I work out the mean difference across 96 trials in 10 separate samples. In order to detect outliers for statistical analysis, ...
3
votes
0answers
845 views

Outliers causing Heywood case in CFA using MLM estimator in lavaan

I am trying to perform a CFA on data (10 indicators: $n=300$) that is severely non-normal but continuous (counts of a clinical behavior over a period of weeks): many cases are at zero, a fair few ...
3
votes
0answers
3k views

Residual analysis and diagnostics for GEE Models in R

Some colleagues asked me to perform a residual analysis on both linear models and generalized estimating equation (GEE) models. I know it is a faux-pas in some circles to remove outliers, but in our ...
3
votes
0answers
217 views

Publications discussing “checking to see if outliers affect the result”

I have noticed it is somewhat common for people to collect data, perform a statistical test (e.g., t-test) and then also look for possible "outliers" in the data. Then they remove the outliers and ...
3
votes
0answers
779 views

Outlier detection of an unevenly spaced time series

I found the Rob H answer to this question very interesting and works pretty well. However, I also would like to apply this methodology to an unevenly spaced time series like the following: ...
3
votes
1answer
557 views

Fitting time series with outliers

I have daily sales data for a department store for the past 850 days. I have indicators on the major holidays and the days leading up to the major holidays. The number of days before the holidays that ...

1
2 3 4 5
9