Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

0
votes
1answer
16 views

Get rid of the irrelevant points [on hold]

Red Cloud Take some time to look at the picture above. We can notice a red cloud. There's a red dense cloud and some irrelevant red points around that red cloud. Suppose the red cloud to be the set ...
0
votes
0answers
19 views

How does Outliers affect logistic regression? [duplicate]

I see this is answered here : How does outlier impact logistic regression? But I am not able to understand how does it affect logistic regression , can anyone take a step back and explain in a bit ...
1
vote
1answer
45 views

How to calculate the standard average of a set excluding outliers? [closed]

I have a set of numbers, and I need to calculate their average excluding outlier values (which I don't know a priori). It came to mind that many years ago I studied Standard Deviation. Could I apply ...
21
votes
1answer
2k views

Why is PCA sensitive to outliers?

There are many posts on this SE that discuss robust approaches to Principal Component Analysis (PCA) but I cannot find a single good explanation of why PCA is sensitive to outliers in the first place?
0
votes
0answers
24 views

How to find the outliers in a long-tailed distribution [duplicate]

I have a long-tailed distribution of years where small values have a higher probability to occur than higher values (median ~ 30 years, maximum value ~30'000 years). I want to find the outliers for ...
0
votes
0answers
16 views

Removing an outlier in a single measure in multivariate data

We have recorded some kinematic data, and are looking at three measures derived from the movement data for each subject. In order to identify outliers, we are using the Mahanalobis distance. I got ...
0
votes
0answers
6 views

Is there a function to calculate the critical Tietjen-Moore value?

The NIST article on the Tietjen-Moore Test for outliers recommends calculating the critical value by simulation, generating 10,000 sets of data. Is there a function that can be numerically integrated ...
0
votes
0answers
7 views

Should I remove the outlier with low cook`s distance?

I am performing a normal linear regression with the model with 24 observations, 5 main effect and 3 interaction terms. When fitting the studentized residuals against the fitted value, I spot that the ...
0
votes
0answers
23 views

Should you normalize your training data for Local Outlier Factor

Say I'm using scikits implementation of Local Outlier Factor with euclidean distance being used by the reachability function. My input features are magnitudes apart, so is it advisable that I ...
2
votes
0answers
29 views

For a small dataset, should I apply an adjustment to the 3-sigma rule when using it for outlier detection?

Standard deviations for small datasets (for example, less than 20 data points) are more volatile/higher variance than those for large datasets. If we want to use the 3-sigma rule for outlier detection,...
0
votes
0answers
45 views

Using ARIMA with exogenous regressors for outlier detection in R

I would like to detect outliers in real-time data that is aggregated per hour. For this example, I've selected the hourly pedestrian data from Melbourne, Australia (Pedestrian volume (updated monthly),...
0
votes
1answer
20 views

Is there any paper shown how to solve an one class SVM by SMO type algorithm

The one class SVM can be used as an outlier rejection. An example can be found on. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3041295/ Generally one class SVM is shown as a constrained quadratic ...
1
vote
0answers
23 views

What regression diagnostics should I perform for an ordered probit?

Currently I have done the following diagnostics with the linktest multicollinearity with vif the parallel lines assumption with lr test of the oprobit and goprobit. I have seen that I may have to ...
0
votes
0answers
15 views

Finding outliers in data set with random jumps in values

I am working with a simple dataset of the format date, value. I need to find the outliers in the values of datapoints, but not sure how to approach the situation, as the values naturally have random (...
0
votes
0answers
25 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
0
votes
0answers
36 views

Imputating data without changing the Mahalanobis distance

I have a multi-variable dataset (rows/observations are independent). I want to remove outliers in the data based on Mahalanobis distance (MD). In base R there is already a function which calculates it ...
1
vote
0answers
17 views

Name for spurious linear Regression Plots

Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
0
votes
0answers
33 views

Identifying influential data points (not necessarily outliers) for a LMER model

I am looking for a good way to identify influential data points in a mixed effect model with interaction (since these are NOT necessary outliers). (1) Is there any good function to do that? How would ...
0
votes
0answers
13 views

Finding Thresholds for Sales Outliers

I am trying to determine the best methods for analyzing future data in terms of the prior 5 years worth of sales data. The data is sales related so when I plot out the transactions it is highly skewed ...
0
votes
1answer
26 views

Best practice for outlier removal in Investigating a process deviation

In a controlled process, in which a specific product depicted a deviation in a final product result. The process is time controlled, in which the historical manufacturing experience of the product is ...
0
votes
0answers
11 views

Correct Usage of IsolationForest

I am working on a Regression problem. I want to remove outliers before building a model, and I narrowed down to IsolationForest (scikit-learn implementation) as it is high dimensional data. I got ...
0
votes
1answer
16 views

Clustering data with outliers without ignoring them

I have 2 dimensional data about customer churn,i Clustered them with k-means but because of Outliers the clusters are not homogeneous.actually Outliers are about customers which are very important , ...
2
votes
2answers
60 views

Outlier and correlation

Hi, I have a question. The scatter plot doesn't show any type of correlation and there is an outlier. If the outlier was to be removed, would the correlation: Increase dramatically Increase ...
0
votes
0answers
16 views

Tukey’s rule for outlier analysis: What does it mean when the HIGH is below the mean?

Hi as you know the upper boundary of Tukey's rule is Upper Range = Q3 + (1.5 * IQR) however recently I came across something that does not make sense to me. ...
0
votes
0answers
41 views

What are the different influences of outliers regarding the feature scaling methods: standardization VS. normalization?

I've come to know that normalization (MinMax scaling) and standardization (Z-score normalization) on data have different influences from outliers in the data. In About Feature Scaling and ...
0
votes
0answers
18 views

How to define “top users” of a group

I have N users, ordered by the number of purchases they made in a period of time and I want to find a "scientific" criteria to classify them into normal users and top users, being the top users the ...
2
votes
2answers
117 views

Does the presence of the outliers affect the 1NN algorithm?

I am working on KNN algorithm. I uploaded and prepared the following dataset. ...
3
votes
1answer
148 views

Robust covariance and OGK outlier detection

I'm calculating the robust covariance of a data set in order to use mahalanobis distance for outlier detection. There are few methods to calculate the covariance in the equation. Using the Fast-MCD ...
1
vote
1answer
39 views

LOF (Local Outlier Factor) choosing the value of k

Just want to understand one thing. Let's say for any data set I select k=20 and generate LOF for each point and then I show all the points in the descending order of its LOF. Now when I am analyzing ...
0
votes
1answer
9 views

Repeated measures factorial assessed using planned contrasts

I have dependent data that can be arrayed as a factorial. I understand that although factorials can be assessed through ANOVA, it is not necessary or always desirable to do so. A second option is 1-...
0
votes
0answers
20 views

Outlier treatment in tsoutlier function of forecast package

I am struggling to understand the way tsoutlier function of forecast package identifies the outliers in time series and suggest the replacements.Thanks very much in advance.
1
vote
1answer
43 views

How do I formally test outliers from a linear regression?

Hello (and thanks for reading this). I have a set of data that looks at the area of remaining retina against age. I have three points of data per patient. We know that over time, the area of ...
2
votes
0answers
25 views

How do I formally test outliers from a linear regression?

I have a set of data that looks at the area of remaining retina against age. I have three points of data per patient. We know that over time, the area of remaining retina decreases exponentially. I ...
2
votes
1answer
72 views

Keep eliminating data points until good correlation coefficient is obtained-using Python

I have been trying to find out a way in order to eliminate outliers from a dataset. The outliers are removed the following way: Any value which results into a 10% reduction in R2 value needs to be ...
0
votes
0answers
6 views

Comparing discrete distributions when the sample set contains long streams of unique erroneous observations?

There is a reference dataset which is a discrete count over a set of names which produces a distribution $P$, and a set of names that are entered into a website as usernames which has a distribution $...
0
votes
0answers
22 views

What is the best way to remove outliers in the task of forecasting demand?

I have dataset with over 2500 IDs of products. For each ID I need to remove outliers. I removed them by the condition, that everything what lies over 1.5 * IQR should be deleted, but it seems that ...
1
vote
2answers
26 views

Observations with same feature values but different Classes

Observations with the exact same attribute values should also have the same class value (basically the mapping to learn from data). If this is not the case, then there are not enough attribute ...
0
votes
0answers
25 views

R: detect outliers to find the 'real' weight

I have a question about labeling outliers in my data. My data is weights for several products. Of course, there is a 'true weight' of a product. I wish to find/approach that from my data (on average). ...
0
votes
1answer
49 views

Is it necessary to deal with the outliers if we perform Normalisation on the data?

I am wondering, if it is necessary to remove outliers from the dataset if we perform Normalisation on the data as after Normalisation, all the values will shrink to value between 0 and 1. So, is it ...
0
votes
0answers
18 views

When is it better to exclude outliers and calculate the mean of the data instead of using the median?

I already searched on when to use the mean and median and I often see that median might be better than mean when the data is skewed, ordinal, include outliers, etc.. even tho, this might not be always ...
1
vote
0answers
38 views

Detecting outliers in binary data using Mahalanobis distance

I have a binary vector $X_i$, $i=1...N$ of independent Bernoulli variables with parameters $p_i, \mu_i = p_i, \sigma_i^2 = p_i(1-p_i)$ (which is known) and I'm looking for some sort of tail bound to ...
1
vote
1answer
27 views

How to find a population of healthy products among many samples

A measurement is performed on 100 equal products. Some of these products contain defects. I expect defects to be present on a small number of products, say a total of 10. Now I want to create a ...
0
votes
0answers
130 views

Robust Anomaly Detection Algorithm from Netflix?

I have read a lot about the robust anomaly detection of Netflix which they open sourced as part of their Surus Project (https://github.com/Netflix/Surus). The project anomaly detector is based on the ...
0
votes
1answer
84 views

Normality test and Outlier detection [duplicate]

In this question, I would like to ask two things: outlier detection normality test Details are as follows: I need to detect and remove outliers in my data. Before doing that, I want to test if my ...
0
votes
0answers
27 views

How to find outlier in SVM

Consider we have a binary SVM classifier, classifying event A and event B. WHat is the best possible way to tackle an outlier? For example, if we receive something which is neither A nor B we need to ...
0
votes
0answers
17 views

Detecting the change of distribution in one-dimension time-series data

I have a univariate time-series data X1,X2 ... I know some point-wise outlier detection algorithm for this kind of data, but I have no idea how to detect if the ...
0
votes
1answer
67 views

Greater than 30% outliers in small dataset - what to do? Standard test? Test with outliers removed? Robust statistics?

I have a small-sample dataset representing observations from a longitudinal study. My principal interest is in 'change scores' across three parameters (A, B, C). This requires simple paired t-tests. ...
1
vote
0answers
16 views

How to integrate expert knowledge to outlier detection algorithms?

Suppose I have a dataset of 20 features, X1, X2..X20. ...
0
votes
0answers
31 views

Flag outliers first or conduct multiple imputation first?

I am working with a data set in which the dependent variable, Y, is constructed from three variables (y1, y2, y3) that each have missing data. To address this issue with multiple imputation, I've used ...
0
votes
1answer
136 views

identify level shifts in a time series

I have a time series as follows: I want to identify the locations of level shifts in this time series. Are there R packages available to do the job?