An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
15 views

Outlier detection with data (which has categorical and numeric variables) with R

Scenario I have a project about fraud detection where i need to find outliers by kmeans. I have a dataset about bank credits length of 1000. There are 21 columns (14 categorical, 7 numeric ...
0
votes
0answers
20 views

what is name outlier detection method

I want to find the name of method for outlier detection. this method calculate standard deviation and mean of data set. sample that is not in (mean-sd mean+sd) is an outlier.
3
votes
1answer
34 views

Does MCD estimator suffers from swamping effect?

If there are multiple outliers in the data set, the Mahalanobis distance suffers from masking and swamping effects. In order to rectify this problem, robust estimation of location and scale, such as ...
8
votes
2answers
392 views

Removing outliers from data - maximum number of outliers that you can remove?

I have a couple of outliers in my data and I was wanting to exclude them to see if this changes the results. In you opinion, what is the maximum number of outliers one should restrict themselves to? ...
1
vote
0answers
12 views

Running/Online Outlier Detection?

I get a stream of values online from a potentially non-stationary distribution, and would like to calculate whether a value is "rare" i.e. higher values only occur once in every 100 or 1000 values. It ...
0
votes
0answers
22 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...
0
votes
0answers
5 views

Detecting Outlier Using WEKA [migrated]

I am a beginner in WEKA tool. I have develop one new algorithm for outlier detection, i would like to implement this algorithm using WEKA. but i did not know the basic steps for implementing ...
0
votes
0answers
20 views

Statistical test to assess that a bunch of point diverge from background distribution

I've to set of points. First a set of points representing the background, that are very close to the identity line (slope=1). And a second set of points that seems to diverge from this identity line ...
3
votes
1answer
52 views

Is it possible to seed RANSAC with a given line?

I am analyzing a stream of data and I want to seed every new instance with the best guess output (line) of the previous, so as to eventually converge. Given that Scikit Learn - RANSAC is an iterative ...
1
vote
1answer
58 views

Anomaly detection: multivariate Gaussian distribution

I am trying to do anomaly detection on a heterogeneous dataset (There are unknown groups present in the dataset). I want to try multivariate Gaussian distribution based approach, but I was thinking of ...
0
votes
0answers
26 views

How to determine the lower- and upper- tail cutoff values efficiently?

I have a long vector (~1.000.000 entries) of integers from 1 to 2500 each of which expresses the number of occurrences of some sort of event for a certain user. The data can be illustrated as: ...
5
votes
2answers
208 views

Outlier Detection on skewed Distributions

Under a classical definition of an outlier as a data point outide the 1.5* IQR from the upper or lower quartile, there is an assumption of a non-skewed distribution. For skewed distributions ...
1
vote
0answers
29 views

Quantitative qualification of outliers [closed]

Is there any quantitative method to qualify outliers that would help distinguish junk outliers from information-rich ones? For example, if multiple outliers look alike, there is a greater chance that ...
0
votes
0answers
37 views

Outliers in panel data

I have a panel data with 4 waves. The variable of interest for me is hourly pay which increases on average from wave to wave. I want to drop the observations with hourly pay in the bottom and the top ...
2
votes
1answer
41 views

Relation between $R^2$ and $R^2_{(i)}$

Is there a relation between $R^2$ and $R^2_{(i)}$ (where $R^2_{(i)}$ is the $R^2$ of a regression without the point ith. For example if the ith point is an outlier) without having to recalculate all ...
0
votes
0answers
72 views

Novelty and Outlier Detection in Unsupervised Learning Style

Currently I am looking for some method to do novelty and outlier detection. I found some good example here using scikit-learn (Link1). However, it is based on supervised learning and I believe the ...
0
votes
1answer
28 views

Novelty and Outlier Detection for Multi-label Data

I met a problem of using novelty and outlier detection for my multi-label data. For example, I have got some training data that is not polluted by outliers. However, the training data are with ...
0
votes
0answers
22 views

How to find mobile app usage outliers?

I am doing some analysis on the usage data of my company's mobile app. However, the QA team will do some prod-test at any time, which generates enormous amount of usage data and also obscure the ...
0
votes
0answers
24 views

Dealing with outliers: Clustering [duplicate]

I am working with a dataset in R that I will be doing cluster analysis on and I am trying to determine the best way to deal with the outliers.I have twelve variables and most variables have between ...
1
vote
0answers
12 views

Filtering outliers: maintaining consistency over time with newly added records

Each item in a table has a price and is assigned to a region. I filter out outliers by identifying and deleting the records whose prices are more than 3 standard deviations away from the average for ...
1
vote
1answer
50 views

MAD formula for outlier detection

Does anyone know what is the name of this formula? $$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$ where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
2
votes
0answers
22 views

How to call “Inliers” and “Outliers” in French

I asked this on the French Exchange site, but this is stat related so... How do you say "Inliers" and "Outliers" (as with RANSAC) in French? The Wikipedia article doesn't translate them, but honestly ...
3
votes
1answer
142 views

How to find set of directions in Stahel-Donoho outlyingness measure?

Currently I’m trying to understand and use the Stahel-Donoho outlyngness measure. But unfortunately I’ve got a problem in the part where one is taking the maximum over the set of directions. I found ...
0
votes
0answers
13 views

Determine cause of anomaly

I'm trying to perform some kind of clustering based anomaly detection for time series and it gives me solid results. What I am interested in is - are there any methods to determine cause of anomally? ...
6
votes
2answers
184 views

R t.test … NOT significant anymore

I got very confused while looking at help examples of the t.test function ...
0
votes
0answers
8 views

Characterizing “typical behavior” for events?

I need to build a model to characterize what is typical for a series of events, which in turn will be used to flag atypical events. As an example, think of credit card purchases (how often? what ...
1
vote
0answers
67 views

Multivariate outlier detection for PLS model

I am working with a PLS model (library pls) in R, where I am developing calibration models for NIRS data. I have been using other commercial software before that allowed me to detect outliers based on ...
0
votes
0answers
24 views

Fundamental Issues with Influence weighted resampling for bootstrapped predictions

I have a large database 1mill+ from which it is known that there are many influential points and outliers. I am interested in generating a series of predictions from subsets (1,000+) of the data and ...
3
votes
2answers
273 views

Analyze scatter plot

I want to study the relationship between two variables. I've got the following scatter plot. But now I'm hesitating on what to do with this: Should I check the assumptions of OLS and then use the ...
0
votes
0answers
14 views

R: Analyze scatter plot [duplicate]

I want to study the relationship between twe variables. I've got the following scatter plot. But now I'm hesitating on what to do with this: 1) Should I check the assumptions of OLS and then use ...
1
vote
1answer
87 views

What are some useful robust and scalable approaches towards anomaly detection of a time series data?

What are some useful robust and scalable approaches/methods towards anomaly detection of a time series data? I am mainly looking for some practical approaches carried out using Python, R, Java, etc. ...
4
votes
0answers
64 views

How to quickly identify participants responding randomly to self-report psychometric tests with many items?

Many psychological studies involve getting participants to answer a hundred or more closed ended questions. A standard context would be a personality test with 100 items where each item is answered on ...
0
votes
1answer
22 views

unsupervised clustering with “unclassified” items

I have data (some behavioral features, measured on some scales) on people. I want to cluster people based on these features. This is an unsupervised scenario, as I have no prior knowledge on the ...
2
votes
2answers
80 views

How to find a wrong predictor value based on other correlated predictors

I have five correlated predictors, ref the following pairs plot: Now I suspect that sometimes a predictor is wrong, as these come from different sources. In other words, four of the predictors of ...
0
votes
0answers
39 views

Top coding: Replace outliers by the value of the mean +/- 6 times the standard deviation in SAS

I'm wondering if anyone can give me some help with this.. I'd like to perform a cluster analysis and this method is very sensitive to outliers. Therefore, top coding is used in similar research: ...
1
vote
0answers
28 views

how to determine outliers in sample affected by ascertainment bias

I don't know if this is a really silly question as I'm in no way a statistician and I don't know if this is something that's actually quite rudimentary... Thanks for reading in advance too it got kind ...
8
votes
1answer
247 views

Can we use leave one out mean and standard deviation to reveal the outliers?

Suppose I have normally distributed data. For each element of the data I want to check how many SDs it is away from the mean. There might be an outlier in the data (likely only one, but might be also ...
0
votes
0answers
93 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
4
votes
1answer
87 views

Outlier Detection in Time-Series: How to reduce false positives?

I'm trying to automate outlier detection in time-series and I used a modification of the solution proposed by Rob Hyndman here. Say, I measure daily visits to a website from various countries. For ...
0
votes
0answers
34 views

What effect, if any, do outliers have on mediation analysis with bootstrapping?

I am running a mediation analysis spread over 6 models. Analysis is performed using the PROCESS macro. Each model includes 1 IV, 2 parallel mediators, and 1 DV. In a couple of the IVs, a number of ...
-1
votes
1answer
54 views

Best clustering technique for outlier detection?

I have around 15-20 points every second, and I would like to detect outliers based on -their density along x-axis , that means if I am using k-mean clustering then I specify that in x-direction max ...
0
votes
0answers
27 views

Kalman filter before or after outlier removal?

I am getting radar data points in form of (x,y) coordinate system relative to my position every ms.[around 10-15 data points]. Now, inorder to have better position estimate of the points, I would like ...
2
votes
0answers
39 views

Use cases for a unique metric

So I was thinking about this question: Maximum minus average? This question is on hold while work calls me. When I get a chance I am going to substantially update it. Background: At my previous ...
0
votes
1answer
35 views

Correlation and Outliers

I want to know if someone has some experience working the 'issue' im encountering. I have a series of arrays, 18, time series on a 14 year period, I want to build a correlation matrix with these ...
1
vote
1answer
64 views

Clustering based anomaly detection

I'm trying to implement anomaly detection based on clustering. I'm hopping for confirmation of my approach, and I'm exposing my idea, being aware that I could have miss something in my analysis, so ...
0
votes
2answers
240 views

how to detect outliers from residual plot?

I have the following residual plot. Can I detect outliers from residual plot? I want to remove 200 outliers in my data set, but I do not know how should I do that in R ? residual plots: scatter ...
0
votes
0answers
88 views

Estimating the uncertainty of a bias and a scatter

I have one single set of observational data. Assuming I know the right answer for one property of this data set and then I use one tool to measure this quantity. To get an estimate of the amount of ...
1
vote
0answers
17 views

Factor analysis using “outliers-only” time series

Some background I run a factor analysis of a time series $Y$ using a standard OLS model with n+1 independent variables $(F,X_1...X_n)$, where $F$ is the main factor (from an explanatory power ...
1
vote
0answers
17 views

Sample datasets with known outliers for IQR, Q-test and Z-test [closed]

Is anyone aware of a source for sample data sets with known outliers? I've been looking around for years but haven't come up with a solution, short of creating my own limited database. Sets with ...
0
votes
0answers
13 views

Outlier detection in weighted time series

Given a set of observations $X _n = \{x_1, \dots, x_n\}$, and a new observation $x_{n+1}$, we aim to find whether $x_{n+1}$ is outlier. One approach to solve this problem is to check whether ...