Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

0
votes
0answers
12 views

Finding outliers in data set with random jumps in values

I am working with a simple dataset of the format date, value. I need to find the outliers in the values of datapoints, but not sure how to approach the situation, as the values naturally have random (...
0
votes
0answers
17 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
0
votes
0answers
27 views

Imputating data without changing the Mahalanobis distance

I have a multi-variable dataset (rows/observations are independent). I want to remove outliers in the data based on Mahalanobis distance (MD). In base R there is already a function which calculates it ...
1
vote
0answers
16 views

Name for spurious linear Regression Plots

Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
0
votes
0answers
15 views

Identifying influential data points (not necessarily outliers) for a LMER model

I am looking for a good way to identify influential data points in a mixed effect model with interaction (since these are NOT necessary outliers). (1) Is there any good function to do that? How would ...
0
votes
0answers
12 views

Finding Thresholds for Sales Outliers

I am trying to determine the best methods for analyzing future data in terms of the prior 5 years worth of sales data. The data is sales related so when I plot out the transactions it is highly skewed ...
0
votes
1answer
25 views

Best practice for outlier removal in Investigating a process deviation

In a controlled process, in which a specific product depicted a deviation in a final product result. The process is time controlled, in which the historical manufacturing experience of the product is ...
0
votes
0answers
7 views

Correct Usage of IsolationForest

I am working on a Regression problem. I want to remove outliers before building a model, and I narrowed down to IsolationForest (scikit-learn implementation) as it is high dimensional data. I got ...
0
votes
1answer
13 views

Clustering data with outliers without ignoring them

I have 2 dimensional data about customer churn,i Clustered them with k-means but because of Outliers the clusters are not homogeneous.actually Outliers are about customers which are very important , ...
2
votes
2answers
53 views

Outlier and correlation

Hi, I have a question. The scatter plot doesn't show any type of correlation and there is an outlier. If the outlier was to be removed, would the correlation: Increase dramatically Increase ...
0
votes
0answers
15 views

Tukey’s rule for outlier analysis: What does it mean when the HIGH is below the mean?

Hi as you know the upper boundary of Tukey's rule is Upper Range = Q3 + (1.5 * IQR) however recently I came across something that does not make sense to me. ...
0
votes
0answers
29 views

What are the different influences of outliers regarding the feature scaling methods: standardization VS. normalization?

I've come to know that normalization (MinMax scaling) and standardization (Z-score normalization) on data have different influences from outliers in the data. In About Feature Scaling and ...
0
votes
0answers
18 views

How to define “top users” of a group

I have N users, ordered by the number of purchases they made in a period of time and I want to find a "scientific" criteria to classify them into normal users and top users, being the top users the ...
0
votes
0answers
24 views

Anomaly Detection of Qualitative Variables

I have a data set of the following form: ...
2
votes
2answers
71 views

Does the presence of the outliers affect the 1NN algorithm?

I am working on KNN algorithm. I uploaded and prepared the following dataset. ...
3
votes
1answer
105 views

Robust covariance and OGK outlier detection

I'm calculating the robust covariance of a data set in order to use mahalanobis distance for outlier detection. There are few methods to calculate the covariance in the equation. Using the Fast-MCD ...
1
vote
1answer
14 views

LOF (Local Outlier Factor) choosing the value of k

Just want to understand one thing. Let's say for any data set I select k=20 and generate LOF for each point and then I show all the points in the descending order of its LOF. Now when I am analyzing ...
0
votes
1answer
8 views

Repeated measures factorial assessed using planned contrasts

I have dependent data that can be arrayed as a factorial. I understand that although factorials can be assessed through ANOVA, it is not necessary or always desirable to do so. A second option is 1-...
-1
votes
0answers
11 views

level shift outlier model

Does somebody happen to know how to calculate the forecast with the LS formula since it got denominator? I got confused because of that. Here's the model I've been using for the forecast.
0
votes
0answers
12 views

Outlier treatment in tsoutlier function of forecast package

I am struggling to understand the way tsoutlier function of forecast package identifies the outliers in time series and suggest the replacements.Thanks very much in advance.
1
vote
1answer
33 views

How do I formally test outliers from a linear regression?

Hello (and thanks for reading this). I have a set of data that looks at the area of remaining retina against age. I have three points of data per patient. We know that over time, the area of ...
2
votes
0answers
21 views

How do I formally test outliers from a linear regression?

I have a set of data that looks at the area of remaining retina against age. I have three points of data per patient. We know that over time, the area of remaining retina decreases exponentially. I ...
2
votes
1answer
72 views

Keep eliminating data points until good correlation coefficient is obtained-using Python

I have been trying to find out a way in order to eliminate outliers from a dataset. The outliers are removed the following way: Any value which results into a 10% reduction in R2 value needs to be ...
0
votes
0answers
6 views

Comparing discrete distributions when the sample set contains long streams of unique erroneous observations?

There is a reference dataset which is a discrete count over a set of names which produces a distribution $P$, and a set of names that are entered into a website as usernames which has a distribution $...
0
votes
0answers
18 views

What is the best way to remove outliers in the task of forecasting demand?

I have dataset with over 2500 IDs of products. For each ID I need to remove outliers. I removed them by the condition, that everything what lies over 1.5 * IQR should be deleted, but it seems that ...
1
vote
2answers
24 views

Observations with same feature values but different Classes

Observations with the exact same attribute values should also have the same class value (basically the mapping to learn from data). If this is not the case, then there are not enough attribute ...
0
votes
0answers
17 views

R: detect outliers to find the 'real' weight

I have a question about labeling outliers in my data. My data is weights for several products. Of course, there is a 'true weight' of a product. I wish to find/approach that from my data (on average). ...
0
votes
1answer
47 views

Is it necessary to deal with the outliers if we perform Normalisation on the data?

I am wondering, if it is necessary to remove outliers from the dataset if we perform Normalisation on the data as after Normalisation, all the values will shrink to value between 0 and 1. So, is it ...
0
votes
0answers
17 views

When is it better to exclude outliers and calculate the mean of the data instead of using the median?

I already searched on when to use the mean and median and I often see that median might be better than mean when the data is skewed, ordinal, include outliers, etc.. even tho, this might not be always ...
1
vote
0answers
10 views

SPC charts - Do these determine signifcant variation?

My understanding of SPC charts is that they simply pin point when a datum is outside of 3 Z scores (dependent on how you define your confidence intervals). When a data point is outside of these ...
1
vote
0answers
23 views

Detecting outliers in binary data using Mahalanobis distance

I have a binary vector $X_i$, $i=1...N$ of independent Bernoulli variables with parameters $p_i, \mu_i = p_i, \sigma_i^2 = p_i(1-p_i)$ (which is known) and I'm looking for some sort of tail bound to ...
1
vote
1answer
27 views

How to find a population of healthy products among many samples

A measurement is performed on 100 equal products. Some of these products contain defects. I expect defects to be present on a small number of products, say a total of 10. Now I want to create a ...
0
votes
0answers
41 views

Robust Anomaly Detection Algorithm from Netflix?

I have read a lot about the robust anomaly detection of Netflix which they open sourced as part of their Surus Project (https://github.com/Netflix/Surus). The project anomaly detector is based on the ...
0
votes
1answer
61 views

Normality test and Outlier detection [duplicate]

In this question, I would like to ask two things: outlier detection normality test Details are as follows: I need to detect and remove outliers in my data. Before doing that, I want to test if my ...
0
votes
0answers
17 views

How to find outlier in SVM

Consider we have a binary SVM classifier, classifying event A and event B. WHat is the best possible way to tackle an outlier? For example, if we receive something which is neither A nor B we need to ...
0
votes
0answers
13 views

Detecting the change of distribution in one-dimension time-series data

I have a univariate time-series data X1,X2 ... I know some point-wise outlier detection algorithm for this kind of data, but I have no idea how to detect if the ...
0
votes
1answer
47 views

Greater than 30% outliers in small dataset - what to do? Standard test? Test with outliers removed? Robust statistics?

I have a small-sample dataset representing observations from a longitudinal study. My principal interest is in 'change scores' across three parameters (A, B, C). This requires simple paired t-tests. ...
1
vote
0answers
15 views

How to integrate expert knowledge to outlier detection algorithms?

Suppose I have a dataset of 20 features, X1, X2..X20. ...
0
votes
0answers
21 views

Flag outliers first or conduct multiple imputation first?

I am working with a data set in which the dependent variable, Y, is constructed from three variables (y1, y2, y3) that each have missing data. To address this issue with multiple imputation, I've used ...
0
votes
1answer
47 views

identify level shifts in a time series

I have a time series as follows: I want to identify the locations of level shifts in this time series. Are there R packages available to do the job?
1
vote
0answers
64 views

Cooks distance and categorical features

I try to detect outliers by cook´s distance for a regression. If I only use numeric features as explanatory variables it works fine. However, if I add categorical features to the explanatory features ...
1
vote
0answers
56 views

Median absolute deviation outlier detection for new data points

I have been reading this article for outlier detection and the method proposed works really well when applied over my data set. However, it would be great if i could run outlier check for new data ...
2
votes
2answers
33 views

How to test difference in means (due to outliers) when the distribution is not normal?

Background I have two conditions: A and B, with around 400 measurement points for each condition. The most common result for both conditions is around '700', and neither one has a result below '600'....
1
vote
2answers
42 views

Should we identify outliers from population prior to taking sample?

I am revising undergrad statistics course via this course, where i am learning technique to pull out sample from population. While ensuring that sample is decent representative of population, i am ...
1
vote
1answer
80 views

Why does Uniform Distributions have no outliers?

I was reading about the relation between Kurtosis and Outliers on Wiki and got across a line 'An example of a platykurtic distribution is the uniform distribution, which does not produce outliers' [...
1
vote
1answer
33 views

anomaly detection for payment industry's 3 dimension data

Assuming there is a time series dataset of 3 columns: userid, timestamp, transaction_amount. There are millions of users. Years of data. What might be an easy and fast algo/ models to detect anomalies ...
2
votes
0answers
38 views

Using bootstrap for robust estimation

I am hoping to understand the process of bootstrapping outlier-contaminated data, and the effects on (simple) OLS estimators. In particular, we have a DGP, $$Y_t = \beta X_t + \epsilon_t$$ where $\...
0
votes
0answers
21 views

identifying multiD outliers

suppose you have two columns of data, one composed of numbers very close to 1 and numbers very close to 3, and the same for the ...
1
vote
1answer
96 views

Robust Principal Component Analysis for Anomaly Detection

I went through the algorithm and some papers for the Robust PCA and although i understood it, that a matrix M is composed of a lower rank matrix L and a sparse matrix S. If i'm correct it is the ...
0
votes
2answers
39 views

Correlated regressors in linear regression

I have a sample of 412 young subjects, measured twice in an interval between 20 days and 3 years. I am interested in how two external factor (lets say sunlight and ice-cream) relates to growth. ...