An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
13 views

The outlier in non-random data

I have tested a time series data by using Run Test. The result shows that it is non random. How could I detect which data point(s) caused that non-randomness? Is there have any methods I can use? ...
0
votes
2answers
45 views

One Class SVM strange decision boundary

I am trying to plot the decision boundary of a One Class SVM. This is a 2 dimensional representation of my training data And here the picture of the prediction obtained on the training data ...
0
votes
0answers
34 views

How to represent outliers for multi dimensional data (local outlier factor)

Below graph taken from http://en.wikipedia.org/wiki/Local_outlier_factor displays "LOF scores : LOF image : This is great for two dimensional data but what about data > than two dimensions. How ...
0
votes
0answers
6 views

Mislabeled training instance detection and relabeling

I have some text data represented by sparse BOWs features ( ~ 5k features). This data must be classified into (~20) categories, however my training labels data appear to be very noisy (> 20 % wrong ...
0
votes
0answers
21 views

Time-series detection algorithm for multi-seasonal data using Python

My data: I have two seasonal patterns in my hourly data... daily and weekly. For example... each day in my dataset has roughly the same shape based on hour of the day. However, certain days like ...
0
votes
0answers
14 views

How can I reach the plot of outlier effetc of a TC outlier?

In this article there is a plot of the outlier effect. Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format? How can I have this ...
2
votes
3answers
64 views

What is “the shortest half of the data”?

Here is a histogram (realized with JMP) displaying two types of box plot called outlier box plot and quantile box plot. Right below, there are a bunch of explanations of the meaning of the ...
0
votes
0answers
33 views

Finding transformation function for a distribution that looks like exponential

Suppose that we have two data sets, R and P. R is larger than or equal to ...
3
votes
2answers
225 views

Outliers and the mean

I would like to know what the following example is called in mathematics: In a gymnastics competition the judges scored a competitor as 10, 8, 3, 7, 7, 9, and 8. I recall that the ending score was ...
1
vote
0answers
47 views

Transform long tail data set to bell curve [closed]

I'm analyzing a data set that has an extremely long tail, and I'm looking for a way to transfer the data into a bell curve so I can apply statistical analysis to it. Hope this makes sense, any help ...
3
votes
1answer
35 views

Identifying outliers for within-patient numerical data

I have a simple dataset of people's heights, many people with measurements on multiple days (once a year for 10 years, say). I have the date of each measurement. Some of the height values are ...
0
votes
0answers
17 views

Bootstrapping in SAS - PROC LOGISTIC - Next steps ? how to score / perform diagnostics?

My question is as follows. I am referencing the following paper by David Cassell - wherein David talks about bootstrapping techniques in SAS using PROC SURVEYSELECT (many thanks to David - truly a ...
12
votes
4answers
741 views

Good form to remove outliers?

I'm working on statistics for software builds. I have data for each build on pass/fail and elapsed time and we generate ~200 of these/week. The success rate is easy to aggregate, I can say that 45% ...
0
votes
1answer
33 views

Outlier detection using k-means in a binary classification problem

I'm using k-means in every class of a binary classification problem and remove samples that have high distance from center of my features (21 features so 21 ...
1
vote
1answer
27 views

Leverage - influential points

Do we look at the absolute value of the leverage or the relative value? For instance, based on the chart below, the largest leverage is about 0.023, it is big compared to other data points, but I'm ...
1
vote
0answers
27 views

Order of preprocessing steps in a binary classification problem

I have these stages (ordered) for preprocessing in my binary classification problem. Dividing data based on criteria (class1 and class2 databases) Outlier ...
1
vote
0answers
41 views

Outlier Removal in Non-Normal Data

I am currently working on a project in which I have to eliminate outliers from non-normally distributed data sets. The data sets are subsets of a fairly large database (order of millions of ...
0
votes
1answer
25 views

Is my understanding of how to calculate the reachability distance in local outlier factor correct?

Reading lof implementation at : http://www.cse.ust.hk/~leichen/courses/msc-it5210/lectures/LOF_Example.pdf the local reachability distance is given as : I don't fully understand this equation as ...
2
votes
1answer
19 views

Is Outlier detection in two separate databases is equal to one combined database?

Suppose that we have two databases : Database_1 and Database_2 . Database_1 has 300 samples ...
1
vote
0answers
18 views

How to pick the “sigma” when sigma clipping?

In astronomy (and I'm sure in other places as well) it is very common to use sigma-clipping as the outlier rejection scheme. The idea is that you regress the data, subtract the fit from the points, ...
0
votes
1answer
16 views

Use random forest outliers to detect group of variables

I have a input data and an output binary variable . The y value is 1 if the patient get ill. ...
0
votes
0answers
28 views

Publication bias - trim and fill - macro/syntax

I'm looking for an SPSS or Excel macro/syntax that will allow me to compute Duval and Tweedie’s trim and fill procedure (see paper here: ...
1
vote
1answer
83 views

Outlier detection in out-sample data for the purpose of classification

This is the most widely used method for outlier detection in econometrics and statistical problems. X is our data that we're searching for outliers in it (in ...
0
votes
2answers
58 views

Can decision tree be used for fraud detection in this way?

A large dataset with more than 100 variables including a target variable. A small portion of target = 1 cases are fraud or due to other errors. I want to identify these target = 1 cases, i.e. fraud or ...
1
vote
0answers
27 views

High dimensional model estimation with outliers

I have a set H of k m-dimensional hyperplanes in n dimensional space, where ...
5
votes
1answer
78 views

Detecting outliers in circular data?

I've a day of the year data about number of event occurence in different sites: A day of the year is circular data. I know that a usual detecting of outliers, for example by boxplot is no use here: ...
1
vote
1answer
41 views

Analyzing regression results

I have done a regression model where i determine the number of cubes (independent variable) based on the amount of units i started with for each product type (dependent variables, ...
1
vote
1answer
25 views

Have conducted a bivariate correlation - want to identify the outlier - is there a 'statistical' way to do this?

I run correlation (on SPSS) between a governance indicator 1996 and the same governance indicator in 2012. As would be expected there is a near perfect correlation. What I am interested in, however, ...
2
votes
1answer
16 views

Small number of points driving correlation

I am currently looking at trying to find high correlations in a series of data. I initially just thought I could check the correlation of the data and choose those with the highest correlation but ...
1
vote
0answers
14 views

How to do multivariate outlier detection in mixed data with category?

I have a data table where the entries are in the following format. The first column is category, which represent the product category. I have 5 such categories. ...
2
votes
2answers
85 views

Iterative process for removing extreme samples

My samples follow heavy tail distributions. I use a process to detect and remove "extreme" samples that goes like this: Measure mean and standard deviation of samples. Remove samples higher than ...
2
votes
1answer
150 views

Detect outliers in very small data set

I have a data set that includes the different response times of a user that is visiting a web application. For example, a visitor enters www.test.com in the browser and navigates through this domain ...
0
votes
0answers
26 views

Anomaly detection for one feature vector

I have a $n$-dimensional vector of ordered multiple testing $p$-values and I would like to reject the first values that are under a certain threshold $\alpha$. I am looking at this problem as an ...
6
votes
1answer
326 views

Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format?

Comments: Firstly I would like to say a big thank you to the author of the new tsoutliers package which implements Chen and Liu's time series outlier detection which was published in the Journal of ...
2
votes
1answer
37 views

Find rows in data that are statistically different from the mean

I have the following data for various event locations. Each event can either be a success or failure (binary values). Thus the mean = percentage of successes. The data represent the history of events ...
2
votes
2answers
84 views

Outlier problems

Having built a regression model with an ordinal response variable and predictors comprised of categorical and continuous nature, I have some questions that pertain to one of the final goals, i.e. ...
0
votes
1answer
52 views

Identifying subsets for outlier detection in local outlier factor

I am trying to gain better understanding of the idea of local outliers (as discussed in this pdf) and how the function is implemented. Here are the key passages from the pdf: Local outliers: ...
4
votes
2answers
111 views

Outlier detection using regression

Can regression be used for out lier detection. I understand that there are ways to improve a regression model by removing the outliers. But the primary aim here is not to fit a regression model but ...
7
votes
1answer
171 views

Robust estimation of kurtosis?

I am using the usual estimator for kurtosis, $\hat{K}=\frac{\hat{\mu}_4}{\hat{\sigma}^4}$, but I notice that even small 'outliers' in my empirical distribution, i.e. small peaks far from the center, ...
1
vote
0answers
18 views

What's the proper way to do automatic iterative outlier rejection?

I know this is a touchy subject, so I'll proceed with caution... I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
0
votes
1answer
43 views

Linear Regression with Outlier accounting in Bugs

I'm trying to redo an exercise in BUGS from this webpage: a linear regression over a data set with some outliers, using a model that accounts for them. This model uses a mixture of signal and noise ...
0
votes
1answer
180 views

Outlier detection using clustering and dissimilarity matrix in R

I have some problems in finding the outliers using clustering. The data.frame is ~20000 observations and each row has mixed types of variables(numeric, nominal and binary). What I want to do is to ...
2
votes
0answers
53 views

How to detect outliers with longitudinal data?

I am running a pooled OLS and Random Effects (RE) model and I would like to test for whether there are any outliers. I know how to do this for OLS, but I just dont know how to do it for Random ...
2
votes
2answers
38 views

How to appropriately represent certain outliers

I am working with a dataset representing a material's 'Range of Coverage', or, a calculated amount of time it is expected to stay in stock. This calculation is based on a material's usage during a ...
11
votes
2answers
112 views

using neighbor information in imputing data or find off-data (in R)

I have dataset with assumption that nearest neighbors are best predictors. Just a perfect example of two-way gradient visualized- Suppose we have case where few values are missing, we can easily ...
0
votes
0answers
86 views

Outlier detection with ROBPCA for multivariate poisson/non-normal data

It is stated here[1] that we can use ROBPCA to detect outliers for multivariate data. After reading the manual ([2] page 12 : "multivariate normal model etc."), I think the ROBPCA method is also ...
0
votes
0answers
39 views

Outliers in a bimodal distribution

Somewhat related to this post. However my question is a little specific. I have some data from gene expression quantification from single cells using RNA sequencing. I haven't had a look at it but I ...
2
votes
0answers
23 views

Von Mises distribution to detect outliers

I am working out the difference between two angles from a circle, and I work out the mean difference across 96 trials in 10 separate samples. In order to detect outliers for statistical analysis, ...
3
votes
3answers
295 views

Anomaly detection: what algorithm to use?

Context: I'm developing a system that analyzes clinical data to filter out implausible data that might be typos. What I did so far: To quantify the plausibility, my attempt so far was to normalize ...
2
votes
0answers
52 views

Can Chauvenet's criterion be used with non-normal data?

Can I use Chauvenet's criterion on set of observations where a normal distribution cannot be assumed?