An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

3
votes
1answer
29 views

Identifying outliers for within-patient numerical data

I have a simple dataset of people's heights, many people with measurements on multiple days (once a year for 10 years, say). I have the date of each measurement. Some of the height values are ...
0
votes
0answers
11 views

Bootstrapping in SAS - PROC LOGISTIC - Next steps ? how to score / perform diagnostics?

My question is as follows. I am referencing the following paper by David Cassell - wherein David talks about bootstrapping techniques in SAS using PROC SURVEYSELECT (many thanks to David - truly a ...
12
votes
4answers
604 views

Good form to remove outliers?

I'm working on statistics for software builds. I have data for each build on pass/fail and elapsed time and we generate ~200 of these/week. The success rate is easy to aggregate, I can say that 45% ...
0
votes
1answer
27 views

Outlier detection using k-means in a binary classification problem

I'm using k-means in every class of a binary classification problem and remove samples that have high distance from center of my features (21 features so 21 ...
1
vote
1answer
25 views

Leverage - influential points

Do we look at the absolute value of the leverage or the relative value? For instance, based on the chart below, the largest leverage is about 0.023, it is big compared to other data points, but I'm ...
1
vote
0answers
26 views

Order of preprocessing steps in a binary classification problem

I have these stages (ordered) for preprocessing in my binary classification problem. Dividing data based on criteria (class1 and class2 databases) Outlier ...
1
vote
0answers
27 views

Outlier Removal in Non-Normal Data

I am currently working on a project in which I have to eliminate outliers from non-normally distributed data sets. The data sets are subsets of a fairly large database (order of millions of ...
0
votes
1answer
21 views

Is my understanding of how to calculate the reachability distance in local outlier factor correct?

Reading lof implementation at : http://www.cse.ust.hk/~leichen/courses/msc-it5210/lectures/LOF_Example.pdf the local reachability distance is given as : I don't fully understand this equation as ...
2
votes
1answer
19 views

Is Outlier detection in two separate databases is equal to one combined database?

Suppose that we have two databases : Database_1 and Database_2 . Database_1 has 300 samples ...
1
vote
0answers
16 views

How to pick the “sigma” when sigma clipping?

In astronomy (and I'm sure in other places as well) it is very common to use sigma-clipping as the outlier rejection scheme. The idea is that you regress the data, subtract the fit from the points, ...
0
votes
1answer
15 views

Use random forest outliers to detect group of variables

I have a input data and an output binary variable . The y value is 1 if the patient get ill. ...
0
votes
0answers
25 views

Publication bias - trim and fill - macro/syntax

I'm looking for an SPSS or Excel macro/syntax that will allow me to compute Duval and Tweedie’s trim and fill procedure (see paper here: ...
1
vote
1answer
64 views

Outlier detection in out-sample data for the purpose of classification

This is the most widely used method for outlier detection in econometrics and statistical problems. X is our data that we're searching for outliers in it (in ...
0
votes
2answers
53 views

Can decision tree be used for fraud detection in this way?

A large dataset with more than 100 variables including a target variable. A small portion of target = 1 cases are fraud or due to other errors. I want to identify these target = 1 cases, i.e. fraud or ...
1
vote
0answers
26 views

High dimensional model estimation with outliers

I have a set H of k m-dimensional hyperplanes in n dimensional space, where ...
4
votes
0answers
55 views

Detecting outliers in circular data?

I've a day of the year data about number of event occurence in different sites: A day of the year is circular data. I know that a usual detecting of outliers, for example by boxplot is no use here: ...
1
vote
1answer
34 views

Analyzing regression results

I have done a regression model where i determine the number of cubes (independent variable) based on the amount of units i started with for each product type (dependent variables, ...
1
vote
1answer
25 views

Have conducted a bivariate correlation - want to identify the outlier - is there a 'statistical' way to do this?

I run correlation (on SPSS) between a governance indicator 1996 and the same governance indicator in 2012. As would be expected there is a near perfect correlation. What I am interested in, however, ...
2
votes
1answer
15 views

Small number of points driving correlation

I am currently looking at trying to find high correlations in a series of data. I initially just thought I could check the correlation of the data and choose those with the highest correlation but ...
1
vote
0answers
12 views

How to do multivariate outlier detection in mixed data with category?

I have a data table where the entries are in the following format. The first column is category, which represent the product category. I have 5 such categories. ...
2
votes
2answers
84 views

Iterative process for removing extreme samples

My samples follow heavy tail distributions. I use a process to detect and remove "extreme" samples that goes like this: Measure mean and standard deviation of samples. Remove samples higher than ...
2
votes
1answer
144 views

Detect outliers in very small data set

I have a data set that includes the different response times of a user that is visiting a web application. For example, a visitor enters www.test.com in the browser and navigates through this domain ...
0
votes
0answers
26 views

Anomaly detection for one feature vector

I have a $n$-dimensional vector of ordered multiple testing $p$-values and I would like to reject the first values that are under a certain threshold $\alpha$. I am looking at this problem as an ...
4
votes
1answer
247 views

Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format?

Comments: Firstly I would like to say a big thank you to the author of the new tsoutliers package which implements Chen and Liu's time series outlier detection which was published in the Journal of ...
2
votes
1answer
36 views

Find rows in data that are statistically different from the mean

I have the following data for various event locations. Each event can either be a success or failure (binary values). Thus the mean = percentage of successes. The data represent the history of events ...
2
votes
2answers
77 views

Outlier problems

Having built a regression model with an ordinal response variable and predictors comprised of categorical and continuous nature, I have some questions that pertain to one of the final goals, i.e. ...
0
votes
1answer
42 views

Identifying subsets for outlier detection in local outlier factor

I am trying to gain better understanding of the idea of local outliers (as discussed in this pdf) and how the function is implemented. Here are the key passages from the pdf: Local outliers: ...
4
votes
2answers
103 views

Outlier detection using regression

Can regression be used for out lier detection. I understand that there are ways to improve a regression model by removing the outliers. But the primary aim here is not to fit a regression model but ...
7
votes
1answer
169 views

Robust estimation of kurtosis?

I am using the usual estimator for kurtosis, $\hat{K}=\frac{\hat{\mu}_4}{\hat{\sigma}^4}$, but I notice that even small 'outliers' in my empirical distribution, i.e. small peaks far from the center, ...
1
vote
0answers
17 views

What's the proper way to do automatic iterative outlier rejection?

I know this is a touchy subject, so I'll proceed with caution... I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
0
votes
1answer
42 views

Linear Regression with Outlier accounting in Bugs

I'm trying to redo an exercise in BUGS from this webpage: a linear regression over a data set with some outliers, using a model that accounts for them. This model uses a mixture of signal and noise ...
0
votes
1answer
154 views

Outlier detection using clustering and dissimilarity matrix in R

I have some problems in finding the outliers using clustering. The data.frame is ~20000 observations and each row has mixed types of variables(numeric, nominal and binary). What I want to do is to ...
2
votes
0answers
47 views

How to detect outliers with longitudinal data?

I am running a pooled OLS and Random Effects (RE) model and I would like to test for whether there are any outliers. I know how to do this for OLS, but I just dont know how to do it for Random ...
2
votes
2answers
37 views

How to appropriately represent certain outliers

I am working with a dataset representing a material's 'Range of Coverage', or, a calculated amount of time it is expected to stay in stock. This calculation is based on a material's usage during a ...
11
votes
2answers
111 views

using neighbor information in imputing data or find off-data (in R)

I have dataset with assumption that nearest neighbors are best predictors. Just a perfect example of two-way gradient visualized- Suppose we have case where few values are missing, we can easily ...
0
votes
0answers
74 views

Outlier detection with ROBPCA for multivariate poisson/non-normal data

It is stated here[1] that we can use ROBPCA to detect outliers for multivariate data. After reading the manual ([2] page 12 : "multivariate normal model etc."), I think the ROBPCA method is also ...
0
votes
0answers
35 views

Outliers in a bimodal distribution

Somewhat related to this post. However my question is a little specific. I have some data from gene expression quantification from single cells using RNA sequencing. I haven't had a look at it but I ...
2
votes
0answers
22 views

Von Mises distribution to detect outliers

I am working out the difference between two angles from a circle, and I work out the mean difference across 96 trials in 10 separate samples. In order to detect outliers for statistical analysis, ...
3
votes
3answers
245 views

Anomaly detection: what algorithm to use?

Context: I'm developing a system that analyzes clinical data to filter out implausible data that might be typos. What I did so far: To quantify the plausibility, my attempt so far was to normalize ...
2
votes
0answers
48 views

Can Chauvenet's criterion be used with non-normal data?

Can I use Chauvenet's criterion on set of observations where a normal distribution cannot be assumed?
4
votes
0answers
45 views

Outlier detection in beta distributions

Say I have a large sample of values in $[0,1]$. I would like to estimate the underlying $\text{Beta}(\alpha, \beta)$ distribution. The majority of the samples come from this assumed ...
13
votes
3answers
908 views

How to assess skewness from a boxplot?

How to decide skewness by looking at a boxplot built from this data: 340, 300, 520, 340, 320, 290, 260, 330 One book says, "If the lower quartile is farther from the median than the upper quartile, ...
1
vote
1answer
42 views

Recognize outliers in a set as data collected

Suppose I am aggregating data at multiple granularities, where each key is associated with the number of counts collected during the time interval. EX: 1 minute { x:2, y:5, z:3, a:312 } This is a ...
3
votes
1answer
68 views

How to interpet residual plot?

As a part of a design of experiments course I'm taking, I ran an experiment at home. The experiment was checking how water boiling time changes under certain factors (5 overall factors) all which had ...
0
votes
0answers
23 views

Can splines produce a plot with more than one fixed effect? If so, how can I correct for a fixed effect?

I have my statistician supervisor wanting me to plot a series of graphs using splines. On the x axis I have volume of milk, on the y axis days. Now, he says I need to "correct for age and birth ...
0
votes
0answers
36 views

standard deviation set for spline cubic regression model

I'm attaching two sets I ran in SAS, regarding spline cubic regression models. I'm new to stats and trying my best to learn but I'm stuck here. How do I get SAS to select outliers based on two or 3 ...
1
vote
0answers
57 views

When to remove/replace outliers for different types of analysis

I have a general question about handling outliers when doing univariate and multivariate analysis, but I'm going to present my specific situation for discussion clarity. I'm dealing with ecological ...
2
votes
1answer
33 views

What is an indicator in statistics and why is it used to refine models?

In a problem we are asked to "refine the fitted model by using an indicator for the outlier". What does it mean to use an indicator for an outlier?
0
votes
0answers
71 views

Detecting outliers in non parametric data

what is the best (and easy) way to detect outliers in spss for skewed, not exactly normally distributed data? Thanks a lot
0
votes
1answer
31 views

Outliers and Linearity for EFA

I am having 56 likert scaled items for IV and 28 likert scaled items for dv. As to fulfill the assumption for EFA, outliers and linearity need to be checked. Can anyone tells me what method/analysis ...