An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

-1
votes
1answer
22 views

Outliers detection for clustering methods

I'm in the middle of a result analysis for some clustering methods, doing quality tests for different clustering outputs coming from a singular input dataset where data preprocessing and cleaning ...
0
votes
0answers
31 views

When removing outlier is right? Removing techniques for outliers in R [duplicate]

I was searching outlier removal in R and I saw some comments related to almost never you should remove outlier from dataset. I wonder when we should remove outlier? I have a dataset consisting ...
1
vote
0answers
24 views

Removing outliers based on cook's distance in R Language

I have this R code for linear regression: fit <- lm(target ~ age+sales+income, data = new) How to identify influential observations based upon cook's distance ...
0
votes
0answers
26 views

Label outliers for anomaly detection

I am trying to detect anomalies using unsupervised learning techniques. However, I have the problem that it is impossible to generate controlled anomalies to use as a test set. My idea is to discover ...
2
votes
2answers
90 views

What to do if residuals are not normally distributed?

I was wondering what to do with the following non-normal distribution of residuals of my multiple regression. Normality test of standardized residual ...
0
votes
1answer
27 views

Representing signals as feature vectors for deviation detection

I want to monitor (automatic-)gearbox failures on some vehicles. For each vehicle I have a captured signal representing the selected gear at each one millisecond (the values are discrete between 0 and ...
1
vote
0answers
16 views

a crossover repeated measure feed trial using R

I attempt to analyse a crossover feed trial using R but I`m not good at statistical analysis. My data contains 10 subjects, equally divided into 2 groups named A and B. Each group were fed with diet A ...
3
votes
4answers
65 views

How to plot algorithm runtime for huge input set?

Fo my bachelor thesis, I want to compare the runtime of two algorithms. The runtime is measured by letting these algorithms run for every value in a huge input set. This input set can be partitioned ...
1
vote
1answer
77 views

outlier detection

I have a question regarding outlier detection. The dataset consists of monthly data for each location. So, for example, the first location "USA" will have values of [8,1,2,1,0] from Jan to May, ...
2
votes
4answers
168 views

How to detect abnormality in an otherwise very systematic and regular time-series data for temperature measurement?

I have time-series data, let's say a pandas series, with time (sampling frequency is hourly) as its index and temperature measurement across that time. I want some statistical/time-series principle ...
0
votes
1answer
7 views

Detecting outlying distributions of ratio data

I have a dataset consisting of hundreds of repeat observations on thousands of agents. Each observation is a ratio between two distance measures, A and B, where A is always larger than B. Thus, my ...
1
vote
0answers
9 views

Need help understanding this algorithm for a robust estimator for Geom. dist

I am trying to figure out a way to estimate the parameter for a Geometric distribution, using a random sample that is influenced by outliers. Searching through previous questions/answers, I saw this: ...
1
vote
1answer
44 views

Using percentiles and inter-quartile-range for outlier detection in skewed data

I am analyzing the age of a certain group of people and I want to use percentiles and inter-quartile-range in the data to flag possible outliers. I am getting Q1 - 25th percentile, Q3 - 75th ...
0
votes
0answers
35 views

R - Multivariate Gaussian Outlier Detection

I have a dataset with N samples (>200) and 5 variables. In my implementation I need to classify some samples after a "calibration". This calibration is done by using an amount of the dataset (i.e. 100 ...
1
vote
1answer
30 views

Rationale behind iterating standard deviation after removing outliers [duplicate]

I'll preface this by saying that I haven't taken a stats class yet, so talk to me like a five year old. If it matters, the data I'm working with are execution times for a program and I'm trying to ...
1
vote
0answers
10 views

Given a data table with outlier data points, how to determine whether the data table include global outliers or local outliers?

As we know, outliers are categorized into (i) global outliers and (ii) local outliers. Local outliers are outliers comparing to their local neighborhoods, instead of the global data distribution. ...
0
votes
0answers
33 views

Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data

I am working on a data analysis project over the summer. The main goal is to use some access logging data in the hospital about user accessing patient information and try to detect abnormal accessing ...
0
votes
1answer
32 views

Outlier detection with ARIMA models?

I have several different time series with monthly values for 8 years, where I fit an ARIMA model. And the purpose is to forecast the next year and indicate possible outliers in a fancy way. Is the ...
0
votes
0answers
28 views

Can we improve the Multiple R-square(coefficient of determination) value for a dataset of a linear regression model?

Here is what i am doing. I am building a logarithmic model in linear form based on the correlation between two variables shown in the graph! lm(y~logx,data=logdata) -- i have only one predictor and ...
2
votes
1answer
44 views

Anomaly detection based on clustering

I understand that there a lot of different methods for anomaly detection, based on classification, clustering, nearest neighbors, statistical, etc. I'm trying out clustering based approach. So, I'm ...
0
votes
0answers
25 views

How can you detect outliers in a group of face images?

I'm trying to filter an image database which contains some irrelevant pictures. All the faces are labeled with points around the face contour, eyes, mouth, eyebrows, have age and gender. The faces are ...
0
votes
2answers
60 views

R - Multivariate K-nearest neighbor outlier detection

I'm trying to implement the algorithm K-nearest neighbor to detect outlier from a multivariate dataset. I don't know how to do it. Could you provide me some example?
0
votes
1answer
38 views

I want to generate outliers in binary logistic model

I want to generate outliers in binary logistic model What I want is: to select 3 elements of 10 elements that are randomly generated, and: if the selected value is 1 convert it to zero & if ...
3
votes
1answer
62 views

Correct procedures to detect and correct outliers for aggregated/SKU time series

Background I am currently working with sets of product sales time series at SKU-level for a FMCG company. Data are available in a weekly format for multiple years and sales data for hundreds of ...
4
votes
1answer
70 views

Anomaly detection in time series data

Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I ...
2
votes
0answers
28 views

Logistic regression: Absolute values for P

I am stuck with a problem (actually two problems). I have a dataset of about 150 cases and 30 or so dichotonous (yes/no) parameters. I selected 6 parameters (after literature study and crosstabs) for ...
1
vote
0answers
23 views

outlier / anomaly detection in high frequency time series data

I am collecting stats from a number of different sensors on a racing car. They update every millisecond and are plotted to a real-time graph. I can see the graph update an observe trends and ...
0
votes
1answer
23 views

What is the problem with statistical outlier detection approaches if we have distribution of attributes?

A group of outlier discovery methods are statistical approaches. Two drawbacks mentioned for statistical methods in many books and papers: They can apply just on a single attribute We need to know ...
0
votes
0answers
22 views

What if we use mean and standard deviation in Stahel-Donoho outlier measure?

I need to use an outlyingness measure in an optimization problem which is already complex. So I need a simple measure of outlyingness. I didn't find any except Stahel-Donoho outlyingness measure. In ...
4
votes
2answers
69 views

Distribution of “sample” mahalanobis distances

Let $x_1,\dots,x_n$ be i.i.d. observations from $N_p(0,\Sigma)$. Let $\hat S=\frac1n\sum_{i=1}^n x_ix_i^T$ be the sample covariance of the samples. Recall that the Mahalanobis distance is defined: ...
4
votes
2answers
101 views

Best statistics to show outliers beyond 2 SD

I try to perform scheduling for an organization based on client flows. We are very seasonal. Our senior management in Washington, DC make us base our schedule on the mean number of clients we have ...
0
votes
0answers
11 views

Remove contigs after assembly that have improbable coverage

I have assembled a large set of small contigs (comparable method to RADseq). To test if the contigs are assembled properly i mapped my reads back to this assembly. So i calculated coverage using ...
3
votes
1answer
94 views

How to select the 'best' trim value for the mean function?

I'm experimenting with the trim parameter to the mean function, E.g. ...
0
votes
0answers
28 views

What are outlier measures for regression problems?

I want to detect outliers automatically and some how eliminate effect of them in a regression problem. In fact I don't even want to detect outliers. I need to just eliminate or minimize the effect of ...
0
votes
0answers
30 views

A strategy to find outliers in a fitted Poisson distribution

I'm using SciPy to fit Poisson distribution to some empirical data in order to find possible misfits. The thing is there are no built-in tools to find outliers in the SciPy kit and I've got no will to ...
0
votes
1answer
35 views

Is there an advantage to using moving average versus removing outliers?

I have a dataset and for each hour there is 3 readings (sometimes missing and sometimes clearly an outlier). I am trying to find the mean of the entire dataset for the parameter. It has been suggested ...
0
votes
1answer
26 views

Whats it called when you fit a linear regression to data with outliers at the end point that influence your regression

So if you have outliers in the middle of your sample it doesn't influence your regression much but if they are at either end of your sample they do.
2
votes
0answers
49 views

Automatic classification of outliers

I have the following plot of data: and I am trying to separate the main part of the data with the outliers that are far away from the main data (for example the data found at around x=250, around ...
3
votes
3answers
129 views

eliminating outliers in MARS regression

I using the regression method called MARS, in R is it called earth and is located in the ...
0
votes
0answers
34 views

anomaly detection with Markov chain

The paper uses a simple technique to detect intrusions in computer systems. I will briefly explain it and ask a question: The paper proposes a simple 1-order Markov chain modelling approach to detect ...
0
votes
2answers
55 views

How to downweigh outlier in a sum?

I have a simple problem. Assume following dataset: resids <- c(,9,8,7,12,14,8,9,15,4,9,10,200) n <- length(resids) p <- 2 Using this dataset I want to ...
3
votes
1answer
47 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
5
votes
1answer
127 views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
2
votes
2answers
44 views

Extreme outlier detection algorithm for erroneous latitudes/longitudes

I have a dataset with latitude/longitude of hotels of a "destination". A destination is a city neighbourhood, whole city, or small region, usually having between 3 and 50 hotels. About 1% of the ...
0
votes
0answers
12 views

Is there a consensus method for defining outliers in a data set? [duplicate]

I am working on a large data matrix and I would like to know if there is a consensus method for defining outliers in a data set? I can 'eye-ball' it on a density plot, but it would be nice not having ...
1
vote
1answer
170 views

Simple algorithm for online outlier detection of a generic time series II: Daily cycle within annual

I have several years of sensor data (temperature and relative humidity) that records every 1/2 hour. When the sensor dies, it often starts throwing bad data mixed in with good data before it dies ...
3
votes
1answer
109 views

Linear regression with violated assumptions

I am trying to find out the determinants of cognitive function. The outcome variable is the mini–mental state examination which is a 30 point questionnaire response that has score values from 0 to ...
1
vote
0answers
54 views

Should outliers in a time series be removed before or after detrending?

I am doing a classical time series analysis. When do I remove outliers in the data? After detrend or before detrend?
0
votes
0answers
44 views

Testing outlier influence on random effects in linear mixed effects models

I have been reading a little bit about diagnostics for linear mixed effects models and have started wondering about how outliers may influence random effects in addition to fixed effects. The paper on ...
2
votes
0answers
41 views

Anomaly Analysis (K-Means) - finding suspicious activities/operators

I am relativly new to the field of data mining and want to make a anomaly detection on transactional retail data. I want to use a simple anomaly detection (kmeans at the moment) for finding suspicious ...