An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
22 views

How to determine the lower- and upper- tail cutoff values efficiently?

I have a long vector (~1.000.000 entries) of integers from 1 to 2500 each of which expresses the number of occurrences of some sort of event for a certain user. The data can be illustrated as: ...
5
votes
2answers
177 views

Outlier Detection on skewed Distributions

Under a classical definition of an outlier as a data point outide the 1.5* IQR from the upper or lower quartile, there is an assumption of a non-skewed distribution. For skewed distributions ...
1
vote
0answers
25 views

Quantitative qualification of outliers [on hold]

Is there any quantitative method to qualify outliers that would help distinguish junk outliers from information-rich ones? For example, if multiple outliers look alike, there is a greater chance that ...
0
votes
0answers
31 views

Outliers in panel data

I have a panel data with 4 waves. The variable of interest for me is hourly pay which increases on average from wave to wave. I want to drop the observations with hourly pay in the bottom and the top ...
2
votes
1answer
37 views

Relation between $R^2$ and $R^2_{(i)}$

Is there a relation between $R^2$ and $R^2_{(i)}$ (where $R^2_{(i)}$ is the $R^2$ of a regression without the point ith. For example if the ith point is an outlier) without having to recalculate all ...
0
votes
0answers
38 views

Novelty and Outlier Detection in Unsupervised Learning Style

Currently I am looking for some method to do novelty and outlier detection. I found some good example here using scikit-learn (Link1). However, it is based on supervised learning and I believe the ...
0
votes
1answer
22 views

Novelty and Outlier Detection for Multi-label Data

I met a problem of using novelty and outlier detection for my multi-label data. For example, I have got some training data that is not polluted by outliers. However, the training data are with ...
0
votes
0answers
21 views

How to find mobile app usage outliers?

I am doing some analysis on the usage data of my company's mobile app. However, the QA team will do some prod-test at any time, which generates enormous amount of usage data and also obscure the ...
0
votes
0answers
24 views

Dealing with outliers: Clustering [duplicate]

I am working with a dataset in R that I will be doing cluster analysis on and I am trying to determine the best way to deal with the outliers.I have twelve variables and most variables have between ...
1
vote
0answers
12 views

Filtering outliers: maintaining consistency over time with newly added records

Each item in a table has a price and is assigned to a region. I filter out outliers by identifying and deleting the records whose prices are more than 3 standard deviations away from the average for ...
1
vote
1answer
42 views

MAD formula for outlier detection

Does anyone know what is the name of this formula? $$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$ where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
2
votes
0answers
22 views

How to call “Inliers” and “Outliers” in French

I asked this on the French Exchange site, but this is stat related so... How do you say "Inliers" and "Outliers" (as with RANSAC) in French? The Wikipedia article doesn't translate them, but honestly ...
3
votes
1answer
122 views

How to find set of directions in Stahel-Donoho outlyingness measure?

Currently I’m trying to understand and use the Stahel-Donoho outlyngness measure. But unfortunately I’ve got a problem in the part where one is taking the maximum over the set of directions. I found ...
0
votes
0answers
13 views

Determine cause of anomaly

I'm trying to perform some kind of clustering based anomaly detection for time series and it gives me solid results. What I am interested in is - are there any methods to determine cause of anomally? ...
6
votes
2answers
177 views

R t.test … NOT significant anymore

I got very confused while looking at help examples of the t.test function ...
0
votes
0answers
8 views

Characterizing “typical behavior” for events?

I need to build a model to characterize what is typical for a series of events, which in turn will be used to flag atypical events. As an example, think of credit card purchases (how often? what ...
1
vote
0answers
57 views

Multivariate outlier detection for PLS model

I am working with a PLS model (library pls) in R, where I am developing calibration models for NIRS data. I have been using other commercial software before that allowed me to detect outliers based on ...
0
votes
0answers
21 views

Fundamental Issues with Influence weighted resampling for bootstrapped predictions

I have a large database 1mill+ from which it is known that there are many influential points and outliers. I am interested in generating a series of predictions from subsets (1,000+) of the data and ...
3
votes
2answers
270 views

Analyze scatter plot

I want to study the relationship between two variables. I've got the following scatter plot. But now I'm hesitating on what to do with this: Should I check the assumptions of OLS and then use the ...
0
votes
0answers
14 views

R: Analyze scatter plot [duplicate]

I want to study the relationship between twe variables. I've got the following scatter plot. But now I'm hesitating on what to do with this: 1) Should I check the assumptions of OLS and then use ...
1
vote
1answer
77 views

What are some useful robust and scalable approaches towards anomaly detection of a time series data?

What are some useful robust and scalable approaches/methods towards anomaly detection of a time series data? I am mainly looking for some practical approaches carried out using Python, R, Java, etc. ...
4
votes
0answers
62 views

How to quickly identify participants responding randomly to self-report psychometric tests with many items?

Many psychological studies involve getting participants to answer a hundred or more closed ended questions. A standard context would be a personality test with 100 items where each item is answered on ...
0
votes
1answer
20 views

unsupervised clustering with “unclassified” items

I have data (some behavioral features, measured on some scales) on people. I want to cluster people based on these features. This is an unsupervised scenario, as I have no prior knowledge on the ...
2
votes
2answers
72 views

How to find a wrong predictor value based on other correlated predictors

I have five correlated predictors, ref the following pairs plot: Now I suspect that sometimes a predictor is wrong, as these come from different sources. In other words, four of the predictors of ...
0
votes
0answers
27 views

Top coding: Replace outliers by the value of the mean +/- 6 times the standard deviation in SAS

I'm wondering if anyone can give me some help with this.. I'd like to perform a cluster analysis and this method is very sensitive to outliers. Therefore, top coding is used in similar research: ...
1
vote
0answers
24 views

how to determine outliers in sample affected by ascertainment bias

I don't know if this is a really silly question as I'm in no way a statistician and I don't know if this is something that's actually quite rudimentary... Thanks for reading in advance too it got kind ...
8
votes
1answer
232 views

Can we use leave one out mean and standard deviation to reveal the outliers?

Suppose I have normally distributed data. For each element of the data I want to check how many SDs it is away from the mean. There might be an outlier in the data (likely only one, but might be also ...
0
votes
0answers
67 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
4
votes
1answer
71 views

Outlier Detection in Time-Series: How to reduce false positives?

I'm trying to automate outlier detection in time-series and I used a modification of the solution proposed by Rob Hyndman here. Say, I measure daily visits to a website from various countries. For ...
0
votes
0answers
17 views

What effect, if any, do outliers have on mediation analysis with bootstrapping?

I am running a mediation analysis spread over 6 models. Analysis is performed using the PROCESS macro. Each model includes 1 IV, 2 parallel mediators, and 1 DV. In a couple of the IVs, a number of ...
-1
votes
1answer
52 views

Best clustering technique for outlier detection?

I have around 15-20 points every second, and I would like to detect outliers based on -their density along x-axis , that means if I am using k-mean clustering then I specify that in x-direction max ...
0
votes
0answers
20 views

Kalman filter before or after outlier removal?

I am getting radar data points in form of (x,y) coordinate system relative to my position every ms.[around 10-15 data points]. Now, inorder to have better position estimate of the points, I would like ...
2
votes
0answers
38 views

Use cases for a unique metric

So I was thinking about this question: Maximum minus average? This question is on hold while work calls me. When I get a chance I am going to substantially update it. Background: At my previous ...
0
votes
1answer
31 views

Correlation and Outliers

I want to know if someone has some experience working the 'issue' im encountering. I have a series of arrays, 18, time series on a 14 year period, I want to build a correlation matrix with these ...
1
vote
1answer
51 views

Clustering based anomaly detection

I'm trying to implement anomaly detection based on clustering. I'm hopping for confirmation of my approach, and I'm exposing my idea, being aware that I could have miss something in my analysis, so ...
0
votes
2answers
165 views

how to detect outliers from residual plot?

I have the following residual plot. Can I detect outliers from residual plot? I want to remove 200 outliers in my data set, but I do not know how should I do that in R ? residual plots: scatter ...
0
votes
0answers
87 views

Estimating the uncertainty of a bias and a scatter

I have one single set of observational data. Assuming I know the right answer for one property of this data set and then I use one tool to measure this quantity. To get an estimate of the amount of ...
1
vote
0answers
16 views

Factor analysis using “outliers-only” time series

Some background I run a factor analysis of a time series $Y$ using a standard OLS model with n+1 independent variables $(F,X_1...X_n)$, where $F$ is the main factor (from an explanatory power ...
1
vote
0answers
17 views

Sample datasets with known outliers for IQR, Q-test and Z-test [closed]

Is anyone aware of a source for sample data sets with known outliers? I've been looking around for years but haven't come up with a solution, short of creating my own limited database. Sets with ...
0
votes
0answers
13 views

Outlier detection in weighted time series

Given a set of observations $X _n = \{x_1, \dots, x_n\}$, and a new observation $x_{n+1}$, we aim to find whether $x_{n+1}$ is outlier. One approach to solve this problem is to check whether ...
0
votes
0answers
26 views

Intuition on One Class Support Vector Machines

I am having trouble understanding how one-class SVMs work. They were introduced in a paper by Scholkopf and others (and can be found here). One-class SVMs perform "novelty detection", where a point ...
1
vote
3answers
73 views

ANOVA with outlier group

This is my first question (previously the search function has been enough), so please bear with me. I have a very simple experimental design with one outcome variable and 5 groups. My typical ...
0
votes
0answers
23 views

Any outliers test for any distribution?

I have different data series of rainfall values (mm), and some days there are points out of the normal trend. When I apply multiple distributions to try to fit the series they are different ...
1
vote
0answers
23 views

Handling outliers in the target variable

I'm using a support vector regression model. I know the target variable has some outliers and modeling the data directly leads to bad results (Rsquare close to 0.2). I'm pretty sure the outliers are ...
0
votes
0answers
15 views

The outlier in non-random data

I have tested a time series data by using Run Test. The result shows that it is non random. How could I detect which data point(s) caused that non-randomness? Is there have any methods I can use? ...
1
vote
2answers
76 views

One Class SVM strange decision boundary

I am trying to plot the decision boundary of a One Class SVM. This is a 2 dimensional representation of my training data And here the picture of the prediction obtained on the training data ...
0
votes
0answers
54 views

How to represent outliers for multi dimensional data (local outlier factor)

Below graph taken from http://en.wikipedia.org/wiki/Local_outlier_factor displays "LOF scores : LOF image : This is great for two dimensional data but what about data > than two dimensions. How ...
0
votes
0answers
12 views

Mislabeled training instance detection and relabeling

I have some text data represented by sparse BOWs features ( ~ 5k features). This data must be classified into (~20) categories, however my training labels data appear to be very noisy (> 20 % wrong ...
0
votes
0answers
65 views

Time-series detection algorithm for multi-seasonal data using Python

My data: I have two seasonal patterns in my hourly data... daily and weekly. For example... each day in my dataset has roughly the same shape based on hour of the day. However, certain days like ...
0
votes
0answers
15 views

How can I reach the plot of outlier effetc of a TC outlier?

In this article there is a plot of the outlier effect. Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format? How can I have this ...