All Questions
1,365 questions
1
vote
2
answers
162
views
Checking for an increase in outliers over time
I've been asked to test if there has been an increase in the number and size specifically of the high outliers over the years. The purpose is to show that there are more and higher extreme cases as ...
- 29
7
votes
4
answers
387
views
Can you remove outliers if they are less than 10% of the datapoints? [duplicate]
I am currently attending my first data analysis class and we do some simple hypothesis tests like t test etc. Our teacher told us that we can remove outliers, as long as they are not more that the 10% ...
- 71
2
votes
3
answers
41
views
Testing forecasting accuracy - outliers [ with example]
I have a simple model that produces forecast values. The model works on hourly data. Now, I am only interested in observations with flags. I would like to identify where the forecasts are ...
- 79
0
votes
1
answer
15
views
How can I filter outliers in data that is manually recorded?
Different people have to write down values on a certain type of parameter in order to fill out a table, and people obviously tend to write wrong. Sometimes, by a factor of 1000. This creates a lot of ...
- 1
0
votes
0
answers
30
views
Understanding heuristic-based outlier detection: concerns about scoring, weighting, and validity
I am trying to understand the mathematics and methodology behind a newly published outlier detection algorithm in the Computer & Security journal. This algorithm uses heuristic-based approaches, ...
- 445
2
votes
1
answer
105
views
Finding outliers in mostly zero data
Background
I'm working on an algorithm to find a short pieces of DNA sequence in a long DNA sequence. I won't go in detail of how it actually works, but let me more formally state it to provide ...
- 231
1
vote
0
answers
29
views
How can I identify the distribution of a series of Mahalanobis distances?
If my dataset follows a multivariate t-distribution, what is the cdf of the Mahalanobis distance of a datapoint outside the sample? In other words, if I want to calculate the probability that a ...
1
vote
1
answer
85
views
Local Outlier Factor for time series
I hope this makes sense. I have discovered LOF and tried it in R. However, since I am dealing with time series, the neighbors cannot be "future" neighbors of the current observation(s). I am ...
- 307
0
votes
0
answers
20
views
Looking for a book regarding outlier/anomaly detection in time series using confidence or prediction interval
I read a book/paper long time ago where the author describes how to use model and its confidence/prediction interval to detect outliers and replace them with the forecasting value.
It had a nice time ...
0
votes
0
answers
15
views
Latent variable demonstration with only 3 variables
I collected data for anxiety (ANX), depression (DEP), and posttraumatic stress syndrome (PTSD) symptoms. Spearman's correlation results are the following (...
- 161
1
vote
0
answers
10
views
How to know which features contribute the most to the outlier score after applying GMM detector?
I have a dataset with 100+ features, upon which I test GMM to detect anomalies. For example, I add some Gaussian noise to 5-6 features of 100 points. GMM detects the points easily, but the next ...
- 11
0
votes
0
answers
14
views
Quantifying the Discrepancy Between Two Distributions with Sensitivity to Outliers
I have two probability distributions over a sample space $x \in \Omega$, denoted by $P: \Omega \mapsto \mathbb{R}$ and $Q: \Omega \mapsto \mathbb{R}$. These distributions arise in the context of ...
- 101
2
votes
1
answer
70
views
MSE gets better but $R^2$ gets worse
Consider the following small dataset (around 569 data points), where Uptake is the regression target:
As you can see, most of the variables are skewed, with some of them having only 2 or 3 data ...
1
vote
0
answers
12
views
Determining the multiplier in limits for spotting Outliers
I want to determine the chance of having above-the-expected sales orders for products, then i could use this (my gut feeling and other business analysis) to determine if i should (or not) keep safety ...
- 11
0
votes
0
answers
19
views
Bayesian model missing outliers at cutoff in data
I am having trouble getting the model to fit. I have ED50 values of chlorophyll in corals during a heating experiment. I have 4 reef sites and 4 species of coral with ~14 corals per site-species group....
- 11
0
votes
0
answers
17
views
Computing trends on data with missing value
I am collecting some feedback from my users. There is no guarantee that all users will offer feedback. So in every time period, there will be a lot of feedback forms with no feedback value.
I want to ...
- 101
3
votes
1
answer
248
views
Utilising Paired T-test but data is not normally distributed and there are outliers
I have a data sample of 190 but I have a few outliers and my data is not normally distributed. I intend to use paired T-test to evaluate the pre-post treatment over time. What should I do?
In addition,...
- 69
5
votes
1
answer
357
views
Why does modified z-score not pick up an obvious outlier?
looking to draw on some of your wisdom around modified z-scores as used for detecting outliers.
As far as I can tell from my research, when a distribution might not be normal (e.g. skewed), a modified ...
- 53
0
votes
1
answer
23
views
Outlier detection for data set comparison
I have two data sets with similar columns, one numerical and the rest categorical.
col_1= categorical: city_name,
col_2= categorical: company_name,
col_3 = categorical: product_name,
col_4 = numerical ...
- 1
-1
votes
1
answer
56
views
Usefulness of p-value to flag outliers in a data set [closed]
Suppose I have a set of data such that $$y= a\times x + b + \varepsilon $$
I am trying to find $a$ and $b$, but some $y$'s are outliers and up to 80% of the data is missing, so I don't have access to $...
- 1
2
votes
1
answer
69
views
How can I show statistically that one of my replicates is likely contaminated?
I have a dataset that looks like the below: five replicate samples, each of which is composed of 4 different fractions that sum to 100%. The fifth sample clearly looks visually distinct from the other ...
- 288
0
votes
0
answers
40
views
Determining the p-value of a test statistic, which is not distributed according to a commonly known distribution under the null hypothesis
Currently I am working in R on a project that aims to identify Dragon King events (massive outliers) in large datasets. These outliers appear for example in the city sizes in England, where London is ...
1
vote
1
answer
46
views
What is the interpretation of outlier-robust principal component analysis?
There's a set of methods called "robust" principal component analysis (here, "robust" means resistant to influence from outliers). One example is Hubert et al., "ROBPCA: A new ...
- 1,413
1
vote
1
answer
270
views
to determine the appropriate threshold of the z-score for the non-normally distributed data
I am interested in CPI. And I need to identify outliers in the series. For that, my instructor mentioned about the number of standard deviations from the mean that a data point is. This is Z-score.
I ...
- 1,152
2
votes
2
answers
487
views
Methods for Detecting outliers in a time series
I have a question on detecting the outliers in a time series like PPI, CPI, inflation,...etc.)
Which method should I use? How can I precisely detect these outliers in a test or a method?
Please ...
2
votes
1
answer
45
views
Calculate the confidence that the data point is NOT explained by the regression
I have $n$ independent variables $x_i$ and dependent variables $y_i$ with uncertainties for both $x$ and $y$. I did a linear regression to get a model $\hat y = \beta x$.
Now I want to use this ...
- 155
1
vote
0
answers
102
views
How to deal with outliers in panel data? [closed]
When we have cross-sectional data, we can easily detect and remove outliers. But how should one approach outliers when we are dealing with panel data? Since we have $i$ entities and $t$ times periods, ...
- 345
3
votes
1
answer
71
views
Interpreting Mass-Volume as an evaluation criterion for unsupervised anomaly detection
I have found this paper How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? by
Nicolas Goix that talks about evaluation of unsupervised anomaly scoring functions by the use of ...
- 349
1
vote
1
answer
62
views
Intensity outliers/anomalies in 2D plot
I wonder what kind of method better to use to see outliers on z value of 2D plot. For example, I have measurements of x and y values both in range of 1 to 16 with step of 1. Next I calculate how many ...
- 11
9
votes
4
answers
957
views
Why divide data into 4 parts for IQR, and not into parts of 20 or 10 percentages each?
Why divide data into 4 parts for IQR, versus into more parts, such as 20 or 10 percent per part?
I know that interquartile range by definition means 25%, but that is not my question.
I think that ...
- 99
2
votes
1
answer
110
views
Maximum likelihood estimation with (robust) Huber-White standard errors appropriate for outlier management?
Is maximum likelihood estimation with robust Huber-White standard errors and a scaled test statistic — which is asymptotically equal to the Yuan-Bentler test statistic — appropriate for data with ...
- 247
0
votes
0
answers
16
views
How to compute a weighted mean as a measure of central tendency?
Imagine I have data points in triplicates, as follows
$$
L=\begin{pmatrix}
p_{11} & p_{12} & p_{13}\\
p_{21} & p_{22} & p_{23}\\
& \vdots &\\
p_{n1} & p_{n2} & p_{n3}
\...
- 150
0
votes
0
answers
25
views
Addressing Heteroscedasticity in Mixed Effects Models with glmmTMB and DHARMa in R [duplicate]
I am analyzing ecological data in R, where I aim to understand the impact of urbanization on species trends. My response variable is the coefficient of species trends (estimate), and my main predictor ...
0
votes
0
answers
189
views
How to deal with under-dispersion in negative binomial GLMM?
I have some animal species. I am interested in seeing what is the relationship between the area they occupy (my response variable, p, which is a count of cells) and ...
- 161
4
votes
1
answer
50
views
Why Grubbs test is not finding outliers?
I am new to Statistics. I am trying to find outliers in the given data using Grubbs method in python. But Grubbs is not able to find outliers for my data.
...
- 183
4
votes
1
answer
102
views
Removing high influence points in GAM fits
I am currently building a GAM model to describe house prices. The dataset is a collection of roughly 200K house sale prices $\{P_{i}\}$ together with a vector of house characteristics $\{\textbf{x}_{i}...
- 199
0
votes
1
answer
63
views
Outliers in OLS
I'm running OLS regression in stata with 5-6 IVs (all dummy variables) and a DV (continuous). Does it make sense to check for outliers (e.g., I can't run scatter dv x1 x2) in this case?
Also, is it ...
- 75
0
votes
1
answer
79
views
question about outliers in a 5 term data set
the question states that no data set with only 5 terms has an outlier, and I'm stumped
you're given a set of 5 numbers in ascending order
$$
x_1, x_2, x_3, x_4, x_5
$$
I started by finding the IQR, ...
1
vote
0
answers
66
views
Valid approach: Winsorizing data for main analysis and then doing sensitivity analysis without winsorizing?
I've got a variable with psychological data (N=75) which is distributed pretty symmetrical, but has very few cases with very extreme values, more extreme to the left tail. But nevertheless this data ...
- 11
5
votes
4
answers
863
views
Dropping outlier from linear regression model reducing adjusted R^2
I'm running a linear regression in R on a dataset with 8 independent variables. When I run the model with all variables:
...
- 53
0
votes
0
answers
49
views
How to interpret the cut-off of a Mahalanobis distance in R
Making the cut-off, I see two multivariante outlieres in the plot and get 4 numbers printet in my console.
(43 and 2 in the first row and 2 and 23 in the second row)
What do they stand for? How can I ...
sabrina
0
votes
0
answers
36
views
Approach for multivariate outlier detection when treating missing values with FIML
I‘m calculating a simple regression with one predictor and one dependent variable. Missings treatment is done with full information maximum likelihood (FIML).
Should I do outlier detection, i.e. ...
- 247
2
votes
0
answers
50
views
One-way repeated measures ANOVA with skewed response
We have an experiment with 102 individuals in total. We have an outcome $Y$ (which is a variable related to the structure of a given bone), and we want to know whether this variable $Y$ differs ...
- 359
0
votes
2
answers
55
views
Should you remove outliers in a small insurance dataset? [closed]
Background
I'm working on an insurance dataset that has 4,000 rows, and 30 columns. The target variable is the loss incurred by the customer for the given row.
The target variable's distribution looks ...
- 667
0
votes
0
answers
28
views
What should I do when my data is normal, but not homogen?
My data is (n:43)genotypes with block as replication (n:2). the design is randomized complete block design. and I did normality test and the result said normal, but I did homogeneity test (levenetest) ...
1
vote
1
answer
41
views
Wrong time series detection
I have a problem and I need help.
I have a time series and I need to know if the data is correct. Let me explain with an example. Suppose I have data generated by an atmospheric pressure sensor. The ...
- 141
1
vote
0
answers
28
views
Outliers in Delta Time column, using data from wireshark
I am currently analyzing data downloaded from Wireshark, focusing on real-time network traffic. I need to perform clusterization on this dataset. However, during the visualization process, I noticed ...
- 11
5
votes
2
answers
163
views
Compare failure rates across multiple systems
I work in pharmaceutical manufacturing and one part of a process is a filtration step that uses 'clusters' or 'sets' of single-use (disposable) filters in parallel - the product flows into a manifold ...
- 53
1
vote
1
answer
56
views
How do I handle outliers?
I'm calculating the beta coefficients for some stocks using a single-index linear model with the OLS method. I'm computing the betas at different return intervals to assess the interval effect on the ...
- 131
1
vote
1
answer
73
views
Removing outliers in several groups and for several features
I'm unsure on how to remove or winsorize outliers. Let's say I have 2 groups, treated and control. And I measure feature1 and feature2 for both.
How should I handle outliers? For each group and each ...
- 375