An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

2
votes
0answers
29 views

How to do forecasting with detection of outliers in R? - Time series analysis procedure and Method

I have monthly time series data, and would like to do forecasting with detection of outliers. This is the sample of my data set: ...
0
votes
2answers
19 views

Removing outliers for a more “realistic” linear programming formulation

For the past two school quarters I have been collecting data on my study habits. I would like to use this data to formulate a linear program so that I may optimize my study time. The first quarter I ...
0
votes
0answers
11 views

Question about number of observation in Generalized ESD

According to http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h3.htm The number of observation is denoted by $n-1$ Why dont we just use $n$ instead of $n-1$? Is there any special meaning ...
3
votes
1answer
56 views

How to interpret and do forecasting using tsoutliers package and auto.arima

I have got monthly data from 1993 to 2015 and would like to do forecasting on these data. I used tsoutliers package to detect the outliers, but I do not know how do I continue to forecast with my set ...
0
votes
0answers
16 views

R finding relative maximum from outliers [migrated]

Suppose I have a vector of numbers that I want to find a general cutoff for. For example: x <- c(35, 2, 3, 30, 1, 4, 33, 6, 36) In this case, I would want to ...
2
votes
0answers
26 views

Interpreting linearity in regression when there are outliers

I am trying to determine whether this regression meets all of the assumptions one needs to adhere to when carrying out a multiple linear regression. In looking at the residual plots below, it seems to ...
0
votes
1answer
19 views

Outlier detection using clustering on few rows

I have a frequency table (2 columns) of 20 rows of various transaction amounts. Some of these amounts are fraudulent in nature and are pretty obvious as they appear to be outliers in the scatter ...
2
votes
0answers
15 views

Anomaly detection using principal component classifier, cutoffs selection

I am trying to implement anomaly detection using principal component classifier proposed in "A novel anomaly detection scheme based on principal component classifier" by Shyu et al. It proposes that ...
0
votes
1answer
23 views

Dixon test (Q-test) - different tables found online

I have data from temperature measurements. Unfortunately, sensors tend to have problems and to register unnormally high temperatures. Some cases are easy to detect, others are not, and I have too much ...
1
vote
0answers
23 views

How to account for high-score outliers in predictive sport metrics?

I'm currently trying to rate different teams in a league based on their underlying skill and eventually make predictions on future games using these simple ratings. I have so far found that a team ...
0
votes
0answers
11 views

Methods of Spend Outlier Detection

I'm looking to formalise some logic behind outlier detection for spend categories of survey respondents during a trip or event (e.g. accommodation, meals, shopping, etc). Distributions are zero-heavy, ...
0
votes
0answers
14 views

Rule of thumb for outliers in factor analysis by FactoMineR

I preformed a factor analysis with the FactoMineR package (R) (FAMD function). I would like to examine the results without outliers. What is the rule of thumb in this case? I know one eazy way to ...
1
vote
1answer
30 views

Modify standard deviation with fixed mean

I have a set of pipes that are supposed to receive balanced load from a source. For instance, with 100 of inputs if I have 5 pipes each is supposed to handle 20. But that's not the case in reality and ...
-1
votes
1answer
28 views

Outliers detection for clustering methods

I'm in the middle of a result analysis for some clustering methods, doing quality tests for different clustering outputs coming from a singular input dataset where data preprocessing and cleaning ...
0
votes
0answers
33 views

When removing outlier is right? Removing techniques for outliers in R [duplicate]

I was searching outlier removal in R and I saw some comments related to almost never you should remove outlier from dataset. I wonder when we should remove outlier? I have a dataset consisting ...
1
vote
0answers
51 views

Removing outliers based on cook's distance in R Language

I have this R code for linear regression: fit <- lm(target ~ age+sales+income, data = new) How to identify influential observations based upon cook's distance ...
0
votes
0answers
30 views

Label outliers for anomaly detection

I am trying to detect anomalies using unsupervised learning techniques. However, I have the problem that it is impossible to generate controlled anomalies to use as a test set. My idea is to discover ...
2
votes
2answers
117 views

What to do if residuals are not normally distributed?

I was wondering what to do with the following non-normal distribution of residuals of my multiple regression. Normality test of standardized residual ...
0
votes
1answer
29 views

Representing signals as feature vectors for deviation detection

I want to monitor (automatic-)gearbox failures on some vehicles. For each vehicle I have a captured signal representing the selected gear at each one millisecond (the values are discrete between 0 and ...
1
vote
0answers
24 views

a crossover repeated measure feed trial using R

I attempt to analyse a crossover feed trial using R but I`m not good at statistical analysis. My data contains 10 subjects, equally divided into 2 groups named A and B. Each group were fed with diet A ...
3
votes
4answers
71 views

How to plot algorithm runtime for huge input set?

Fo my bachelor thesis, I want to compare the runtime of two algorithms. The runtime is measured by letting these algorithms run for every value in a huge input set. This input set can be partitioned ...
1
vote
1answer
86 views

outlier detection

I have a question regarding outlier detection. The dataset consists of monthly data for each location. So, for example, the first location "USA" will have values of [8,1,2,1,0] from Jan to May, ...
2
votes
4answers
176 views

How to detect abnormality in an otherwise very systematic and regular time-series data for temperature measurement?

I have time-series data, let's say a pandas series, with time (sampling frequency is hourly) as its index and temperature measurement across that time. I want some statistical/time-series principle ...
0
votes
1answer
7 views

Detecting outlying distributions of ratio data

I have a dataset consisting of hundreds of repeat observations on thousands of agents. Each observation is a ratio between two distance measures, A and B, where A is always larger than B. Thus, my ...
1
vote
0answers
10 views

Need help understanding this algorithm for a robust estimator for Geom. dist

I am trying to figure out a way to estimate the parameter for a Geometric distribution, using a random sample that is influenced by outliers. Searching through previous questions/answers, I saw this: ...
1
vote
1answer
58 views

Using percentiles and inter-quartile-range for outlier detection in skewed data

I am analyzing the age of a certain group of people and I want to use percentiles and inter-quartile-range in the data to flag possible outliers. I am getting Q1 - 25th percentile, Q3 - 75th ...
0
votes
0answers
49 views

R - Multivariate Gaussian Outlier Detection

I have a dataset with N samples (>200) and 5 variables. In my implementation I need to classify some samples after a "calibration". This calibration is done by using an amount of the dataset (i.e. 100 ...
1
vote
1answer
31 views

Rationale behind iterating standard deviation after removing outliers [duplicate]

I'll preface this by saying that I haven't taken a stats class yet, so talk to me like a five year old. If it matters, the data I'm working with are execution times for a program and I'm trying to ...
1
vote
0answers
11 views

Given a data table with outlier data points, how to determine whether the data table include global outliers or local outliers?

As we know, outliers are categorized into (i) global outliers and (ii) local outliers. Local outliers are outliers comparing to their local neighborhoods, instead of the global data distribution. ...
0
votes
0answers
39 views

Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data

I am working on a data analysis project over the summer. The main goal is to use some access logging data in the hospital about user accessing patient information and try to detect abnormal accessing ...
0
votes
1answer
44 views

Outlier detection with ARIMA models?

I have several different time series with monthly values for 8 years, where I fit an ARIMA model. And the purpose is to forecast the next year and indicate possible outliers in a fancy way. Is the ...
0
votes
0answers
32 views

Can we improve the Multiple R-square(coefficient of determination) value for a dataset of a linear regression model?

Here is what i am doing. I am building a logarithmic model in linear form based on the correlation between two variables shown in the graph! lm(y~logx,data=logdata) -- i have only one predictor and ...
2
votes
1answer
59 views

Anomaly detection based on clustering

I understand that there a lot of different methods for anomaly detection, based on classification, clustering, nearest neighbors, statistical, etc. I'm trying out clustering based approach. So, I'm ...
0
votes
0answers
29 views

How can you detect outliers in a group of face images?

I'm trying to filter an image database which contains some irrelevant pictures. All the faces are labeled with points around the face contour, eyes, mouth, eyebrows, have age and gender. The faces are ...
0
votes
2answers
74 views

R - Multivariate K-nearest neighbor outlier detection

I'm trying to implement the algorithm K-nearest neighbor to detect outlier from a multivariate dataset. I don't know how to do it. Could you provide me some example?
0
votes
1answer
43 views

I want to generate outliers in binary logistic model

I want to generate outliers in binary logistic model What I want is: to select 3 elements of 10 elements that are randomly generated, and: if the selected value is 1 convert it to zero & if ...
3
votes
1answer
67 views

Correct procedures to detect and correct outliers for aggregated/SKU time series

Background I am currently working with sets of product sales time series at SKU-level for a FMCG company. Data are available in a weekly format for multiple years and sales data for hundreds of ...
4
votes
1answer
83 views

Anomaly detection in time series data

Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I ...
2
votes
0answers
28 views

Logistic regression: Absolute values for P

I am stuck with a problem (actually two problems). I have a dataset of about 150 cases and 30 or so dichotonous (yes/no) parameters. I selected 6 parameters (after literature study and crosstabs) for ...
1
vote
0answers
24 views

outlier / anomaly detection in high frequency time series data

I am collecting stats from a number of different sensors on a racing car. They update every millisecond and are plotted to a real-time graph. I can see the graph update an observe trends and ...
0
votes
1answer
24 views

What is the problem with statistical outlier detection approaches if we have distribution of attributes?

A group of outlier discovery methods are statistical approaches. Two drawbacks mentioned for statistical methods in many books and papers: They can apply just on a single attribute We need to know ...
0
votes
0answers
24 views

What if we use mean and standard deviation in Stahel-Donoho outlier measure?

I need to use an outlyingness measure in an optimization problem which is already complex. So I need a simple measure of outlyingness. I didn't find any except Stahel-Donoho outlyingness measure. In ...
4
votes
2answers
78 views

Distribution of “sample” mahalanobis distances

Let $x_1,\dots,x_n$ be i.i.d. observations from $N_p(0,\Sigma)$. Let $\hat S=\frac1n\sum_{i=1}^n x_ix_i^T$ be the sample covariance of the samples. Recall that the Mahalanobis distance is defined: ...
4
votes
2answers
107 views

Best statistics to show outliers beyond 2 SD

I try to perform scheduling for an organization based on client flows. We are very seasonal. Our senior management in Washington, DC make us base our schedule on the mean number of clients we have ...
0
votes
0answers
14 views

Remove contigs after assembly that have improbable coverage

I have assembled a large set of small contigs (comparable method to RADseq). To test if the contigs are assembled properly i mapped my reads back to this assembly. So i calculated coverage using ...
3
votes
1answer
113 views

How to select the 'best' trim value for the mean function?

I'm experimenting with the trim parameter to the mean function, E.g. ...
0
votes
0answers
31 views

What are outlier measures for regression problems?

I want to detect outliers automatically and some how eliminate effect of them in a regression problem. In fact I don't even want to detect outliers. I need to just eliminate or minimize the effect of ...
0
votes
0answers
32 views

A strategy to find outliers in a fitted Poisson distribution

I'm using SciPy to fit Poisson distribution to some empirical data in order to find possible misfits. The thing is there are no built-in tools to find outliers in the SciPy kit and I've got no will to ...
0
votes
1answer
41 views

Is there an advantage to using moving average versus removing outliers?

I have a dataset and for each hour there is 3 readings (sometimes missing and sometimes clearly an outlier). I am trying to find the mean of the entire dataset for the parameter. It has been suggested ...
0
votes
1answer
26 views

Whats it called when you fit a linear regression to data with outliers at the end point that influence your regression

So if you have outliers in the middle of your sample it doesn't influence your regression much but if they are at either end of your sample they do.