An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
13 views

How can you detect outliers in a group of face images?

I'm trying to filter an image database which contains some irrelevant pictures. All the faces are labeled with points around the face contour, eyes, mouth, eyebrows, have age and gender. The faces are ...
0
votes
2answers
31 views

R - Multivariate K-nearest neighbor outlier detection

I'm trying to implement the algorithm K-nearest neighbor to detect outlier from a multivariate dataset. I don't know how to do it. Could you provide me some example?
-1
votes
0answers
22 views

R - Outlier detection multivariate in R [duplicate]

I'm looking for few R packages to detect outliers from a multivariate data frame. In the following Dropbox link you can find an example of data. Columns 1 to 5 are the variables and the 6th the label. ...
0
votes
1answer
38 views

I want to generate outliers in binary logistic model

I want to generate outliers in binary logistic model What I want is: to select 3 elements of 10 elements that are randomly generated, and: if the selected value is 1 convert it to zero & if ...
3
votes
1answer
52 views

Correct procedures to detect and correct outliers for aggregated/SKU time series

Background I am currently working with sets of product sales time series at SKU-level for a FMCG company. Data are available in a weekly format for multiple years and sales data for hundreds of ...
3
votes
1answer
48 views

Anomaly detection in time series data

Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I ...
2
votes
0answers
27 views

Logistic regression: Absolute values for P

I am stuck with a problem (actually two problems). I have a dataset of about 150 cases and 30 or so dichotonous (yes/no) parameters. I selected 6 parameters (after literature study and crosstabs) for ...
1
vote
0answers
19 views

outlier / anomaly detection in high frequency time series data

I am collecting stats from a number of different sensors on a racing car. They update every millisecond and are plotted to a real-time graph. I can see the graph update an observe trends and ...
0
votes
1answer
21 views

What is the problem with statistical outlier detection approaches if we have distribution of attributes?

A group of outlier discovery methods are statistical approaches. Two drawbacks mentioned for statistical methods in many books and papers: They can apply just on a single attribute We need to know ...
0
votes
0answers
18 views

What if we use mean and standard deviation in Stahel-Donoho outlier measure?

I need to use an outlyingness measure in an optimization problem which is already complex. So I need a simple measure of outlyingness. I didn't find any except Stahel-Donoho outlyingness measure. In ...
4
votes
2answers
65 views

Distribution of “sample” mahalanobis distances

Let $x_1,\dots,x_n$ be i.i.d. observations from $N_p(0,\Sigma)$. Let $\hat S=\frac1n\sum_{i=1}^n x_ix_i^T$ be the sample covariance of the samples. Recall that the Mahalanobis distance is defined: ...
4
votes
2answers
99 views

Best statistics to show outliers beyond 2 SD

I try to perform scheduling for an organization based on client flows. We are very seasonal. Our senior management in Washington, DC make us base our schedule on the mean number of clients we have ...
0
votes
0answers
8 views

Remove contigs after assembly that have improbable coverage

I have assembled a large set of small contigs (comparable method to RADseq). To test if the contigs are assembled properly i mapped my reads back to this assembly. So i calculated coverage using ...
3
votes
1answer
76 views

How to select the 'best' trim value for the mean function?

I'm experimenting with the trim parameter to the mean function, E.g. ...
0
votes
0answers
27 views

What are outlier measures for regression problems?

I want to detect outliers automatically and some how eliminate effect of them in a regression problem. In fact I don't even want to detect outliers. I need to just eliminate or minimize the effect of ...
0
votes
0answers
20 views

Error message in the pmnorm function for 3 variables

im trying to detect multivariate outliers by calculating the probability of an observation using the multivariate gaussian pdf: This is the "training" data for which the modelparameters (mean and ...
0
votes
0answers
27 views

A strategy to find outliers in a fitted Poisson distribution

I'm using SciPy to fit Poisson distribution to some empirical data in order to find possible misfits. The thing is there are no built-in tools to find outliers in the SciPy kit and I've got no will to ...
0
votes
1answer
34 views

Is there an advantage to using moving average versus removing outliers?

I have a dataset and for each hour there is 3 readings (sometimes missing and sometimes clearly an outlier). I am trying to find the mean of the entire dataset for the parameter. It has been suggested ...
0
votes
1answer
26 views

Whats it called when you fit a linear regression to data with outliers at the end point that influence your regression

So if you have outliers in the middle of your sample it doesn't influence your regression much but if they are at either end of your sample they do.
2
votes
0answers
46 views

Automatic classification of outliers

I have the following plot of data: and I am trying to separate the main part of the data with the outliers that are far away from the main data (for example the data found at around x=250, around ...
3
votes
3answers
115 views

eliminating outliers in MARS regression

I using the regression method called MARS, in R is it called earth and is located in the ...
0
votes
0answers
30 views

anomaly detection with Markov chain

The paper uses a simple technique to detect intrusions in computer systems. I will briefly explain it and ask a question: The paper proposes a simple 1-order Markov chain modelling approach to detect ...
0
votes
2answers
52 views

How to downweigh outlier in a sum?

I have a simple problem. Assume following dataset: resids <- c(,9,8,7,12,14,8,9,15,4,9,10,200) n <- length(resids) p <- 2 Using this dataset I want to ...
3
votes
1answer
43 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
5
votes
1answer
109 views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
2
votes
2answers
43 views

Extreme outlier detection algorithm for erroneous latitudes/longitudes

I have a dataset with latitude/longitude of hotels of a "destination". A destination is a city neighbourhood, whole city, or small region, usually having between 3 and 50 hotels. About 1% of the ...
0
votes
0answers
12 views

Is there a consensus method for defining outliers in a data set? [duplicate]

I am working on a large data matrix and I would like to know if there is a consensus method for defining outliers in a data set? I can 'eye-ball' it on a density plot, but it would be nice not having ...
1
vote
1answer
123 views

Simple algorithm for online outlier detection of a generic time series II: Daily cycle within annual

I have several years of sensor data (temperature and relative humidity) that records every 1/2 hour. When the sensor dies, it often starts throwing bad data mixed in with good data before it dies ...
3
votes
1answer
106 views

Linear regression with violated assumptions

I am trying to find out the determinants of cognitive function. The outcome variable is the mini–mental state examination which is a 30 point questionnaire response that has score values from 0 to ...
1
vote
0answers
53 views

Should outliers in a time series be removed before or after detrending?

I am doing a classical time series analysis. When do I remove outliers in the data? After detrend or before detrend?
0
votes
0answers
32 views

Testing outlier influence on random effects in linear mixed effects models

I have been reading a little bit about diagnostics for linear mixed effects models and have started wondering about how outliers may influence random effects in addition to fixed effects. The paper on ...
2
votes
0answers
34 views

Anomaly Analysis (K-Means) - finding suspicious activities/operators

I am relativly new to the field of data mining and want to make a anomaly detection on transactional retail data. I want to use a simple anomaly detection (kmeans at the moment) for finding suspicious ...
0
votes
0answers
31 views

Outliers in Boxplots when calculating means

I have a dataset for which I am making boxplots. I do not want to include the outliers in the box plot so I give an argument outline=FALSE in the command. In the next step I want to put the mean ...
0
votes
0answers
37 views

Identify outliers with median-absolute-deviation for timeseries data

I am having trouble understanding this particular method of detecting outliers in a time series. Below is the problem: I have a region-of-interest containing 15 voxels. Each voxel contains values ...
3
votes
4answers
243 views

Confused by location of fences in box-whisker plots

In one type of box-whisker plot, the fences at the ends of the whiskers are meant to indicate cutoff values beyond which any point would be considered an outlier. The standard definitions I've found ...
0
votes
1answer
39 views

Outliers and Influential observations in fixed effects regression

I am running a fixed effects regression with a very unbalanced panel data. There are a lot of residuals. Like for half of my observations I get large residuals. So I do not want to simply remove them ...
1
vote
0answers
93 views

Unbalanced Panel data using R - Removing outliers and heteroskedastcity

I am new in R and it’s my first time using it so I’ll appreciate the help. I am estimating income elasticity for electricity consumption using budget shares. I have data for 8 regions categorized into ...
1
vote
0answers
26 views

Identifying outliers from binned data

I have binned data (x-axis) that I've plotted against frequency (y-axis) to see the distribution of the bins and I got this scatterplot:- The bins are arranged from the lowest to the highest. As you ...
6
votes
2answers
118 views

Back-testing or cross-validating when the model-building process was interactive

I have some predictive models whose performance I would like to back-test (i.e., take my dataset, "rewind" it to a previous point in time, and see how the model would have performed prospectively). ...
3
votes
0answers
71 views

How are outliers dealt with in R after detected? [closed]

Once outliers in time series are detected in R how exactly are they dealt with before forecasting? I dont want commands to use i would like the method. Please do not give any answers to do with ...
2
votes
0answers
42 views

what to do with ridiculous but valid leverage points

So I'm having some difficulty fitting a linear model to the data (see other post here glm model fit - can't find a family/link combination that produces good fit). In particular, I'm worried ...
0
votes
0answers
26 views

remove factor level if factor level has outlier in any of other columns in R dataframe

Hello I have a dataframe with one column a factor of patient id's, other columns of continous variables. I want to remove patients from the dataframe if they have an outlier observation in any of the ...
2
votes
3answers
243 views

Putting less weight on certain data points in a series for forecasting

I have a data set that contains outliers (big orders) i need to forecast this series taking the outliers into consideration. I already know what the top 11 big orders are so i dont need to detect them ...
1
vote
0answers
70 views

Is E-Divisive with Medians (the Twitter BreakoutDetection algo) robust and efficient?

There are quite a few algorithms to detect changepoints, outliers, mean shifts, trend shifts etc. out there. Recently I've stumbled upon BreakoutDetection and while it's new and shiny I'd like to know ...
0
votes
2answers
45 views

How to show that a dataset does not contain significant outliers?

I have largish dataset: there is 200 variables and 100 samples. How could I show that the dataset does not contain any significant outliers? All variables have the same unit (millimeters) and have ...
1
vote
0answers
22 views

before clusterisation, should I remove observations with too few measurements?

I have a very unevenly distributed dataset of 462 twitter users. During the window of observation, some of these users have produced as many as 2000 tweets, while others as few as one. My end is to ...
0
votes
0answers
26 views

Fat-tailed data and SVM

Does SVM perform poorly when fat-tailed data with outliers is used? What are some things that could be done to improve learning with such data? Does the choice of kernel and/or kernel parameter ...
0
votes
0answers
26 views

Correlation analysis while detecting outliers

I have simple dataset here. Supposed I want to find out which customers who bought a certain item are more likely to come back after 10 months. I have 2 sets of data The repeat purchase % of users ...
3
votes
0answers
19 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
4
votes
1answer
87 views

Intelligently selecting outliers

I'm trying to remove what might be considered "unreasonable" data by evaluating the percent error in the mean and square root of the variance. Here's the setup: Let's say I have three bids on a ...