Questions tagged [outliers]
An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.
1,326
questions
0
votes
0
answers
4
views
Weighted count of distinct items based on their frequency in a list
I have a list of items. Each item has some properties associated with them. For example, let's consider country. Let's say out of 100 entries, 98 are from US and 2 are from UK. When considering the ...
- 101
0
votes
0
answers
21
views
How to determine a threshold [closed]
I'm struggling with the following problem: we are supposed to determine the best period on which a wind turbine could be stopped to perform programmed maintenance. The data available for the moment is ...
- 101
0
votes
0
answers
19
views
Statistical procedure to remove a time series which is an "outlier" in a set of replicated time series
This data are from measuring optical density in a bacterial growth experiment. These correspond to 4 time series which are biological replicates of exactly the same treatment (with the label 0_4)
The ...
2
votes
2
answers
35
views
Understanding slope in plot of residuals vs. fitted values
How should I interpret the negative relationship between the residual and the fitted value in the plot shown below?
- 21
3
votes
1
answer
32
views
Structural equation modeling (latent growth models): robust estimators to handle outliers?
Can I use robust estimators (e.g., "MLM" and "MLR"estimator lavaan options) to overcome outliers within my sample, or should I remove outliers?
For context, I am modelling the ...
- 115
0
votes
0
answers
11
views
Outliers over multiple time series
I got roughly 400 time-series measurements which are concerned with the sentiment of chat messages from different videos. I want to detect if there are videos which have a higher or lower sentiment ...
- 1
0
votes
1
answer
33
views
Anomaly Detection in Multivariate and Univariate timeseries
I just started exploring Anomaly detection in timeseries for Univariate, Multivariate timeseries.
I read few articles about it, few research papers as well. But every article/research paper has ...
- 19
0
votes
0
answers
6
views
outiler and residual plot [duplicate]
I have a general question. If you have Duration vs Cost graph and you can see some outliers. (let cost be the target variable and duration is a variable) Then, you may need to omit these outliers.
But,...
- 11
0
votes
0
answers
7
views
Outlier detection for observations with sinusoidal relationship, incomplete information
I was looking for recommendations for an outlier rejection problem I find myself needing to solve.
I have a set of measurements (12 at a time) which vary sinusoidally according to
$$
d = \begin{...
- 145
0
votes
0
answers
33
views
Outlier Detection By Dummy Regression Models
The standard procedure for outlier detection in time series implemented in almost all statistical
software tools is based on regression the time series to candidate regression variables. These
...
- 1
0
votes
0
answers
8
views
What test to use to prove significance of deviation from correlation
I have two dependent variables (A and B) and one independent variable (C) for a group of individuals N.
In the correlation of A with C, there seems to be a group of outliers from the main line of the ...
0
votes
0
answers
11
views
Aggregating metrics at parent
I have a model with several independent variables/metrics that each additionally has a corresponding unit, pre-defined upper/lower bounds and a desired value.
Now I wish to compute some basic ...
1
vote
0
answers
34
views
Choosing Outliers using Chauvenet's Criterion
I have normal data and have applied Chauvenet's Criterion to all my data points to determine if there were any outliers that must be removed. I have 2 groups of subjects, male and female, and when I ...
- 53
0
votes
0
answers
14
views
What's wrong with my studentised residual plot? [duplicate]
I have plotted the average SAT score of schools on the x-axis and the studentised residuals on the y-axis.
The code I used to produce this looks like this:
...
- 21
4
votes
2
answers
346
views
Deleting outliers prior to data splitting or only in the training set?
I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
0
votes
1
answer
54
views
Correlation vs Euclidean distance as measures of similarity or closeness between data points with an outlier
I am interested in the comparison of Pearson correlation and Euclidean distance as measures of similarity between data points. Suppose I have 4 data points, w, x, y, z, in a multidimensional space, ...
0
votes
0
answers
40
views
Define and Remove outliers from a Pareto-like Distribution
I currently have a set of data:
1 40950
2 27749
3 14786
.........
15 95
16 75
..........
12205 1
15344 1
19991 1
The first column ...
1
vote
1
answer
64
views
Is there a way to stretch sigmoid function output
I have an array of values and a value that lies outside of array's max value:
arr = [10, 15, 20, 30]
value = 150
and I want to make that value less of an outlier, ...
- 11
2
votes
2
answers
78
views
Is distance from the median instead of from the mean in standard deviation calculation meaningful in case of data with outliers?
I'm borrowing from the idea of Mean of absolute deviation around the median (https://en.wikipedia.org/wiki/Average_absolute_deviation#Mean_absolute_deviation_around_the_median). Would calculating ...
- 21
0
votes
1
answer
19
views
Is there any case in which evaluation of outliers does not render probabilistic sampling deterministic?
As I understand it, probability sampling means that every element in the population has a nonzero probability of being selected as part of the sample. But when some elements have a nonzero probability ...
- 11
0
votes
0
answers
38
views
Anomaly Detection in Categorical Data
I want to build a system that will detect Anomaly for categorical data. I have a timeseries data like this
For metric data these anomalies are calculated
Outliers detection
Trend
Pattern
Change
I ...
- 19
4
votes
2
answers
661
views
Is calculating skewness necessary before using the z-score to find outliers?
For example, if I specify a z-value of 3, then I would look at both sides and know its position in the distribution (99.73%).
Would this change if I have a left or right skewed distribution? Would I ...
- 125
0
votes
0
answers
15
views
Difficulty in identifying outliers beyond limits
I have a various files, all containing data at 1-minute interval. File structure is as follows where D/T represents Date-Time, C1 and C2 are like constants in this task.
...
1
vote
1
answer
61
views
Can I trust the results of a t test on 4 point Likert scale data which hides outliers?
I want to use an unpaired two-sample t-test of random samples of $n=40$ each. The sample data is from 4-point Likert scale assessments. I understand the t-test is not very robust to outliers, which I ...
- 11
3
votes
1
answer
196
views
Why should I split the data when searching for outliers? (pyod)
I am using pyod to detect outliers in data, and I came across this official example: https://github.com/yzhao062/pyod/blob/master/examples/comb_example.py
I have a question regarding the need to split ...
- 125
0
votes
0
answers
25
views
Grubbs test: two/one-sided'nes and degrees of freedom
There are two things which confuse me about the Grubbs-test:
(1) The first aspect involves the two or one-sidednes of the test:
Lets assume I want to check if my W6 dice, where I expect the ...
- 101
0
votes
0
answers
17
views
How can I detect univariate outliers multivariately?
Chose hopefully a catchy title :-) I am looking for a simple algorithm to detect outliers caused by measurement errors
So assune I have given a multivariate sample (30 dimensions) and I want to detect ...
- 1
0
votes
0
answers
50
views
Why do residuals cluster in two group?
I am running a logistic regression in a sample with ~150,000 observations. I am predicting three different outcomes, x, y, and z, that occur in ~10,000, ~4,000, and ~2,000 cases respectively (for each ...
- 21
0
votes
1
answer
40
views
How to identify outliers and drop rows in train splits of all folds, when using StratifiedKFold in GridSearchCV?
For predicting whether a subject has liver disease or not, I'm using StratifiedKFold CV in GridSearch for AdaBoost and RandomForest Classsifiers.
For Outlier anlaysis, I've identified all feature ...
- 1
0
votes
0
answers
64
views
What did Grubbs mean when he "cautioned against interpreting probabilities too literally when normality of the data is not assured"?
In his 1969 paper, Grubbs mentioned that "Until such time as criteria not sensitive to the normality assumption are developed, the experimenter is cautioned against interpreting probabilities too ...
2
votes
1
answer
97
views
Modeling outliers in maximum likelihood estimation with gradient descent
Consider a set of 3D points $X = \{x_1, x_2, ...x_n\} $ with $ x_i\in\mathbb{R}^3$ on which we want to fit an arbitrary probability distribution. The distribution we want to fit models some ...
- 5,576
0
votes
0
answers
13
views
should I check and remove outlier from dataset through multi round
I have a data set as: x1, x2, x3... x9 (non-normal distribution). I check it with box-plot and find one outlier as x9. Normally I will stop then remove x9 and use the new data set. But when I check ...
- 1
0
votes
0
answers
12
views
Systematic/Repeatable Framework for Outlier Removal? [duplicate]
I am using both IQR and Z score > +/- 3SD for outlier detection.
It seems like Z score > +/- 3SD is more strict and yields fewer outliers than IQR, which is better for my purposes (Regression, ...
- 704
0
votes
0
answers
31
views
How to find (and interpret) outliers in set of time series data?
I face with a problem of doing a time series forecasting on multivariate data in the form where different entities have their own 100-day (daily) series of 10 variables, and I'm expected to predict 10-...
- 725
1
vote
0
answers
12
views
shuffling data change OPTICS outlier results
I am trying to use sklearn.cluster.OPTICS to identify outliers, but found an issue:
I use 2 examples with exactly the same data but different orders. They give different results:
1st example
/////////...
- 11
0
votes
0
answers
25
views
What is it called when an outlier falls out of a rolling window statistical calculation?
I have a time series $X_t \sim N(0, 1)$. There is a single outlier at index 347, at 8.5 standard deviations from the mean. If I now compute a rolling window standard deviation of $X_t$ with window ...
- 1,274
0
votes
0
answers
13
views
How to apply dimensionality reduction to a data set with outliers?
I try to apply dimensionality reduction to a multidimensional data set (with numerical features) with significant outliers. I have managed to identify outliers with Isolation Forest but now I'm in a ...
- 725
2
votes
1
answer
79
views
Adjust the "Threshold" in a robust regression
I am trying to perform a robust regressions using the lmrob function in R.
I am getting this error Message:
...
- 23
0
votes
1
answer
57
views
Do outliers begin from or above the whisker-limit? [duplicate]
Does outliers begin on the whisker limit or above it?
In the (Python) example below the calculcated upper whisker limit is 64.8125. Is a value of ...
- 232
4
votes
1
answer
137
views
Should I be concerned about outliers in NB GLMM with an offset term?
I'm working on a negative binomial model for count data. Unfortunately I can't provide a more detailed description because I wasn't explicitly allowed to. All I can say now is that the data is about ...
- 83
0
votes
0
answers
66
views
How to choose threshold value in MCD-based Mahalanobis distances?
A dataset is having 2 or more timeseries
eg: with two timeseries x and y
I need to predict the slope and intercept using Linear regression model b/w x & y.
But my data can have Outliers
My ...
- 19
1
vote
1
answer
21
views
Suggestions on dealing with outliers when sample size is very small AND you must order the results
I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.
For each category we throw out the high and low scores and average the remaining ...
- 11
0
votes
1
answer
63
views
Can high standard deviations explain my non-significant & low effect size results? (please read description)
I'm trying to analyse bullying experiences across three age groups. The DV is scored on a 5-point Likert, and the IV is categorical (ages 11, 13, and 15).
Initially I ran an ANOVA to see if there was ...
- 1
1
vote
0
answers
58
views
Standardization of out-of-sample data
I have a panel (N firms across 10 years) dataset on which I want to estimate and test a prediction model $f$:
\begin{equation}
y = f(x).
\end{equation}
Following common practice, I split my data into ...
- 1,029
0
votes
0
answers
6
views
Time series -- peeking for truncation of outlying values displaced from mean under some structural assumptions
Per https://stats.stackexchange.com/a/204977/384097 the suggestion was made that, given a known distribution, applying a cutoff as means of dealing with outliers is not data leakage. I feel ...
- 1
0
votes
1
answer
84
views
Outlier Detection using OutlierTest
I found an outlier using the outlierTest function in the car package. However, I can see from the results that the Externally Studentized Residual and p-values. This is a result for the full model.
<...
- 11
1
vote
0
answers
25
views
how to find anomalies for a non-normal distribution with seasonality?
I have a time series broken down by day, and there are gaps in it that I have marked in red:
the distribution there is not normal
How do we approach modeling a system that will look for anomalies ...
0
votes
1
answer
135
views
How to deal with Covid outlier in time series/machine learning forecasting?
Disclaimer: I checked some similar questions but I could not find anything in particular that would work for my case.
I am dealing with a time series going from 2015 to 2023. The data points are the ...
- 11
1
vote
0
answers
14
views
How to deal with outliers after heterogeneity test in microarray expression datasets?
I have performed a meta-analysis using five micro-array datasets. After performing meta analysis I visualized the heterogeneity using funnel plot and forest plot (using two up-regulated and two down-...
1
vote
1
answer
108
views
Robustification in lavaan: Difference between M, MV and MVS?
In lavaan, I am running a two-factor CFA on a questionnaire with 28 items, all of which are scored on a 6-point Likert scale. In total I have ~350 participants who completed the questionnaire.
Because ...
- 11