All Questions
1,365 questions
0
votes
0
answers
57
views
Outliers in EDA - With or without?
I'm trying to carry out my first EDA on a Student Performance dataset. The dataset has 395 samples and consists of 33 attributes. After drawing the boxplots and doing some tests I detected outliers in ...
0
votes
1
answer
247
views
Impact of outliers to QQ plot
I'm trying to build an GLM regression (10k samples and 50 dimensions). I ran an analysis of the dependent variable since the regression has a normality assumption for the dependent variable.
The QQ ...
- 53
1
vote
0
answers
35
views
Outlier detection on a measurement stream. Decision theoretic, Bayesian approaches?
I have a stream of real valued measurements $x_1, x_2, \dotsc$ that I expect to be, for the most part, normal distributed with some unknown mean $\mu > 0$ and unknown variance $\sigma^2$. However, ...
- 145
1
vote
0
answers
24
views
Applying Tangent Lines to Log-Scaled Data for Outlier Detection: Seeking Statistical Theories and Models
I've analyzed the view counts for a YouTube channel's videos (just for example), sorting them by views (on the left) and drawing a tangent line to approximate the central trend on a logarithmic scale (...
- 121
4
votes
3
answers
351
views
Outlier detection methods aware of target variable
I am trying to predict ambulance demand for the next hour, for a city area in the USA, based on previous demand, weather, large people gatherings, and similar spatio-temporal factors.
I have noticed ...
- 61
1
vote
1
answer
68
views
Extreme values affecting mean in regression analysis
I am examining whether there is a difference in reaction time (RT) for 2 different conditions (A and B). Participants complete trials in both conditions.
First, I use a linear mixed model to assess ...
- 542
0
votes
1
answer
105
views
Outlier Detection and Removal
I am reading a paper on wind power forecasting and the authors present a plot of the data before outliers are removed and a plot after. However, they don't actually say what method was employed to ...
- 1
2
votes
1
answer
38
views
ARIMA - Identifying an outlier in residuals
I am trying to perform an ARIMA (SARIMAX in fact) and when looking at the residuals I see a large outlier. I am using python statsmodels.tsa.statespace.sarimax.
I ...
- 123
1
vote
1
answer
297
views
Tukey's IQR-method for outliers and highly skewed data
I am writing a thesis on performances on cognitive and linguistic measures. I have used the Tukey IQR method (Q1-1.5*IQR) to detect lower outliers in a non-normally distributed small sample of various ...
- 91
0
votes
1
answer
68
views
Identify outliers in testing data based on trained Gaussian mixture model
I use Gaussian mixture model (GMM) to infer probability density of multidimensional data written as:
$p(x) = \sum_{j=1}^{K}\pi_j*N(x|\bf \mu_j, \Sigma_j)$, where $K$ is a number of mixtures, $\pi_j$ ...
- 11
4
votes
3
answers
351
views
How to compare influence of outlier in regression model. ANOVA of two models in R
I am doing linear regression in R. I have identified an outlier in my data:
...
- 205
1
vote
1
answer
83
views
When detecting outliers for an ANVOA test for multiple groups, should I do this for the whole population sample or by group?
I have data that is grouped by 5 groups. I want to check for outliers as part of a one-way anova. Should the check for outliers on the whole sample, or by group? I am using box plots to look for ...
- 205
1
vote
0
answers
120
views
Evaluate CDF and outliers of multidimensional Gaussian mixture [closed]
I use Gaussian mixture model (GMM) to infer probability density of multidimensional data written as:
$p(x) = \sum_{j=1}^{K}\pi_j*N(x|\bf \mu_j, \Sigma_j)$, where $K$ is a number of mixtures, $\pi_j$ ...
- 11
1
vote
0
answers
31
views
Can there be no outliers as per calculation with IQR formula while the boxplot shows there are outliers in the dataset?
While plotting a box plot, the plot is showing the columns in the dataset has outliers, but while trying to calculate it by IQR formula, it is showing there are 0 outliers in the columns of the ...
- 11
0
votes
1
answer
75
views
Hypothesis testing - Newbie blockers - Update and more
Brief : I'm from manufacturing industry, a processing machine in our production line used to do pressing, polishing and QA one after the other. Now we have a new machine that will perform these at the ...
- 3
1
vote
0
answers
36
views
BEST POSSIBLE WAY to determine significantly high values within zero-inflated univariate continuous distributions
I have more than 50 different distributions, corresponding to 50 different kind of customers, who spend their money in a certain way within a period, being this amount the single variable of interest.
...
- 11
1
vote
0
answers
33
views
Outlier in three dimension [closed]
I am looking for an example of an outlier in three dimensions but that cannot be detected with the three scatters xy, xz, yz. Can anybody help me?
Thanks in advance.
- 21
3
votes
2
answers
62
views
Can using the IQR/Median help with this problem?
I have a set of data in tabular form which records the time it takes 500 people to bake a cake. Each person is assigned a single time: it may take Tom 30 minutes, or Mike 60 minutes to bake a cake.
...
0
votes
0
answers
21
views
Statistical procedure to remove a time series which is an "outlier" in a set of replicated time series
This data are from measuring optical density in a bacterial growth experiment. These correspond to 4 time series which are biological replicates of exactly the same treatment (with the label 0_4)
The ...
2
votes
2
answers
590
views
Understanding slope in plot of residuals vs. fitted values
How should I interpret the negative relationship between the residual and the fitted value in the plot shown below?
- 21
3
votes
1
answer
99
views
Structural equation modeling (latent growth models): robust estimators to handle outliers?
Can I use robust estimators (e.g., "MLM" and "MLR"estimator lavaan options) to overcome outliers within my sample, or should I remove outliers?
For context, I am modelling the ...
- 331
0
votes
1
answer
102
views
Anomaly Detection in Multivariate and Univariate timeseries
I just started exploring Anomaly detection in timeseries for Univariate, Multivariate timeseries.
I read few articles about it, few research papers as well. But every article/research paper has ...
- 33
0
votes
0
answers
11
views
What test to use to prove significance of deviation from correlation
I have two dependent variables (A and B) and one independent variable (C) for a group of individuals N.
In the correlation of A with C, there seems to be a group of outliers from the main line of the ...
1
vote
0
answers
77
views
Choosing Outliers using Chauvenet's Criterion
I have normal data and have applied Chauvenet's Criterion to all my data points to determine if there were any outliers that must be removed. I have 2 groups of subjects, male and female, and when I ...
- 53
5
votes
2
answers
862
views
Deleting outliers prior to data splitting or only in the training set?
I'm working on a dataset with some outliers in the response variable which are actually natural results (not errors). I want to calibrate a model which could then be used to predict on populations ...
2
votes
2
answers
1k
views
Correlation vs Euclidean distance as measures of similarity or closeness between data points with an outlier
I am interested in the comparison of Pearson correlation and Euclidean distance as measures of similarity between data points. Suppose I have 4 data points, w, x, y, z, in a multidimensional space, ...
- 21
0
votes
0
answers
180
views
Define and Remove outliers from a Pareto-like Distribution
I currently have a set of data:
1 40950
2 27749
3 14786
.........
15 95
16 75
..........
12205 1
15344 1
19991 1
The first column ...
1
vote
1
answer
323
views
Is there a way to stretch sigmoid function output
I have an array of values and a value that lies outside of array's max value:
arr = [10, 15, 20, 30]
value = 150
and I want to make that value less of an outlier, ...
- 11
3
votes
2
answers
615
views
Is distance from the median instead of from the mean in standard deviation calculation meaningful in case of data with outliers?
I'm borrowing from the idea of Mean of absolute deviation around the median (https://en.wikipedia.org/wiki/Average_absolute_deviation#Mean_absolute_deviation_around_the_median). Would calculating ...
- 31
0
votes
1
answer
22
views
Is there any case in which evaluation of outliers does not render probabilistic sampling deterministic?
As I understand it, probability sampling means that every element in the population has a nonzero probability of being selected as part of the sample. But when some elements have a nonzero probability ...
- 11
0
votes
0
answers
85
views
Anomaly Detection in Categorical Data
I want to build a system that will detect Anomaly for categorical data. I have a timeseries data like this
For metric data these anomalies are calculated
Outliers detection
Trend
Pattern
Change
I ...
- 33
4
votes
2
answers
1k
views
Is calculating skewness necessary before using the z-score to find outliers?
For example, if I specify a z-value of 3, then I would look at both sides and know its position in the distribution (99.73%).
Would this change if I have a left or right skewed distribution? Would I ...
- 125
1
vote
1
answer
158
views
Can I trust the results of a t test on 4 point Likert scale data which hides outliers?
I want to use an unpaired two-sample t-test of random samples of $n=40$ each. The sample data is from 4-point Likert scale assessments. I understand the t-test is not very robust to outliers, which I ...
- 11
3
votes
1
answer
311
views
Why should I split the data when searching for outliers? (pyod)
I am using pyod to detect outliers in data, and I came across this official example: https://github.com/yzhao062/pyod/blob/master/examples/comb_example.py
I have a question regarding the need to split ...
- 125
0
votes
0
answers
20
views
How can I detect univariate outliers multivariately?
Chose hopefully a catchy title :-) I am looking for a simple algorithm to detect outliers caused by measurement errors
So assune I have given a multivariate sample (30 dimensions) and I want to detect ...
- 1
0
votes
0
answers
126
views
Why do residuals cluster in two group?
I am running a logistic regression in a sample with ~150,000 observations. I am predicting three different outcomes, x, y, and z, that occur in ~10,000, ~4,000, and ~2,000 cases respectively (for each ...
- 21
0
votes
1
answer
59
views
How to identify outliers and drop rows in train splits of all folds, when using StratifiedKFold in GridSearchCV?
For predicting whether a subject has liver disease or not, I'm using StratifiedKFold CV in GridSearch for AdaBoost and RandomForest Classsifiers.
For Outlier anlaysis, I've identified all feature ...
- 1
0
votes
0
answers
87
views
What did Grubbs mean when he "cautioned against interpreting probabilities too literally when normality of the data is not assured"?
In his 1969 paper, Grubbs mentioned that "Until such time as criteria not sensitive to the normality assumption are developed, the experimenter is cautioned against interpreting probabilities too ...
3
votes
1
answer
184
views
Modeling outliers in maximum likelihood estimation with gradient descent
Consider a set of 3D points $X = \{x_1, x_2, ...x_n\} $ with $ x_i\in\mathbb{R}^3$ on which we want to fit an arbitrary probability distribution. The distribution we want to fit models some ...
- 5,696
1
vote
0
answers
21
views
shuffling data change OPTICS outlier results
I am trying to use sklearn.cluster.OPTICS to identify outliers, but found an issue:
I use 2 examples with exactly the same data but different orders. They give different results:
1st example
/////////...
- 11
0
votes
0
answers
31
views
What is it called when an outlier falls out of a rolling window statistical calculation?
I have a time series $X_t \sim N(0, 1)$. There is a single outlier at index 347, at 8.5 standard deviations from the mean. If I now compute a rolling window standard deviation of $X_t$ with window ...
- 1,334
2
votes
1
answer
173
views
Adjust the "Threshold" in a robust regression
I am trying to perform a robust regressions using the lmrob function in R.
I am getting this error Message:
...
- 23
0
votes
1
answer
87
views
Do outliers begin from or above the whisker-limit? [duplicate]
Does outliers begin on the whisker limit or above it?
In the (Python) example below the calculcated upper whisker limit is 64.8125. Is a value of ...
- 282
4
votes
1
answer
538
views
Should I be concerned about outliers in NB GLMM with an offset term?
I'm working on a negative binomial model for count data. Unfortunately I can't provide a more detailed description because I wasn't explicitly allowed to. All I can say now is that the data is about ...
- 93
1
vote
1
answer
76
views
Suggestions on dealing with outliers when sample size is very small AND you must order the results
I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.
For each category we throw out the high and low scores and average the remaining ...
- 11
0
votes
1
answer
374
views
Can high standard deviations explain my non-significant & low effect size results? (please read description)
I'm trying to analyse bullying experiences across three age groups. The DV is scored on a 5-point Likert, and the IV is categorical (ages 11, 13, and 15).
Initially I ran an ANOVA to see if there was ...
- 1
1
vote
0
answers
98
views
Standardization of out-of-sample data
I have a panel (N firms across 10 years) dataset on which I want to estimate and test a prediction model $f$:
\begin{equation}
y = f(x).
\end{equation}
Following common practice, I split my data into ...
- 1,129
0
votes
1
answer
176
views
Outlier Detection using OutlierTest
I found an outlier using the outlierTest function in the car package. However, I can see from the results that the Externally Studentized Residual and p-values. This is a result for the full model.
<...
- 21
1
vote
0
answers
46
views
how to find anomalies for a non-normal distribution with seasonality?
I have a time series broken down by day, and there are gaps in it that I have marked in red:
the distribution there is not normal
How do we approach modeling a system that will look for anomalies ...
0
votes
1
answer
838
views
How to deal with Covid outlier in time series/machine learning forecasting?
Disclaimer: I checked some similar questions but I could not find anything in particular that would work for my case.
I am dealing with a time series going from 2015 to 2023. The data points are the ...
- 11