An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
8 views

remove factor level if factor level has outlier in any of other columns in R dataframe

Hello I have a dataframe with one column a factor of patient id's, other columns of continous variables. I want to remove patients from the dataframe if they have an outlier observation in any of the ...
2
votes
1answer
36 views

Putting less weight on certain data points in a series for forecasting

I have a data set that contains outliers (big orders) i need to forecast this series taking the outliers into consideration. I already know what the top 11 big orders are so i dont need to detect them ...
1
vote
0answers
9 views

Is E-Divisive with Medians (the Twitter BreakoutDetection algo) robust and efficient?

There are quite a few algorithms to detect changepoints, outliers, mean shifts, trend shifts etc. out there. Recently I've stumbled upon BreakoutDetection and while it's new and shiny I'd like to know ...
0
votes
0answers
20 views

R - replace TRUE with actual value [closed]

My goal is to mark the values in plot if the value of jj is less than min_threshold and greater than ...
0
votes
2answers
41 views

How to show that a dataset does not contain significant outliers?

I have largish dataset: there is 200 variables and 100 samples. How could I show that the dataset does not contain any significant outliers? All variables have the same unit (millimeters) and have ...
1
vote
0answers
20 views

before clusterisation, should I remove observations with too few measurements?

I have a very unevenly distributed dataset of 462 twitter users. During the window of observation, some of these users have produced as many as 2000 tweets, while others as few as one. My end is to ...
0
votes
0answers
17 views

Fat-tailed data and SVM

Does SVM perform poorly when fat-tailed data with outliers is used? What are some things that could be done to improve learning with such data? Does the choice of kernel and/or kernel parameter ...
0
votes
0answers
22 views

Correlation analysis while detecting outliers

I have simple dataset here. Supposed I want to find out which customers who bought a certain item are more likely to come back after 10 months. I have 2 sets of data The repeat purchase % of users ...
3
votes
0answers
15 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
4
votes
1answer
70 views

Intelligently selecting outliers

I'm trying to remove what might be considered "unreasonable" data by evaluating the percent error in the mean and square root of the variance. Here's the setup: Let's say I have three bids on a ...
0
votes
1answer
22 views

Spike detection and removal in position data

Is there any good filter to remove big spikes in position data? I think lowpass filter should be good but is it possible to filter 2D position data with assumption its joint distributed? I mean, not ...
1
vote
0answers
8 views

how to determine if a test run in only one geographic area leads to activity different than in other geographic areas

I was recently given data from an experiment that I did not design. In this study, the people running it changed behavior in 1 designative market area (DMA) but not the other DMAs (209 other DMAs). ...
0
votes
0answers
26 views

What do you do with outliers when developing statistical models?

I am a beginner so I have an extremely tough time dealing with outliers. I wanted to ask the community to help me understand rule of thumbs or anythng that would help me deal with these questions ...
0
votes
0answers
16 views

Anomaly Detection with very small number of positives [closed]

I am trying to detect anomalies in a population comprising of 10 features and around 90,000 observations. Past investigations have revealed 18 positives. Given limited data for supervised learning, I ...
0
votes
0answers
38 views

Outside 1.5 times inter quartile range yet not outlier

I have a data set with the following density histogram and box plot. Summary info as given R function. ...
2
votes
1answer
105 views

how to determine skewness from histogram with outliers?

I have the following histogram created in Minitab. I am wondering whether this histogram is actually positively skewed, negatively skewed, or symmetric. By observing the graph itself, it seems that ...
2
votes
1answer
60 views

Elastic net: dealing with wide data with outliers

Recently I was working on a dataset with ~300 observations and 1500 predictors. I used the glmnet package in R to fit an elastic net model, which gave me a ...
6
votes
4answers
139 views

How to prepare/construct features for anomaly detection (network security data)

My goal is to analyse network logs (e.g., Apache, syslog, Active Directory security audit and so on) using clustering / anomaly detection for intrusion detection purposes. From the logs I have a lot ...
0
votes
1answer
35 views

How to predict in advance that a smart meter is failing?

I have an electricity consumption data set collected by smart meters over a year and a half for every hour. The objective is to predict whether the meter could fail earlier than it actually fails.I ...
6
votes
2answers
134 views

What is a good method to identify outliers in exam data?

I give my students an exam that has 8 questions on it. Each question is about a particular topic. The exam is made up on the fly by randomly selecting 1 question for each topic from a pool of ...
2
votes
0answers
56 views

Basic methods for detecting outliers

Let $X$ be a matrix of $n$ rows and $m$ columns. $n$ is the number of samples and $m$ is the number of gene expressions. Gene expressions are basically numerical continuous values. Assume we have a ...
1
vote
1answer
55 views

Detecting anomalies in a time series where new data points will be continuously added

I have a time series data and I will be adding more data points in a consistent manner. I want to figure out whether the new data point added is an outlier, in regards to the previously observed data ...
0
votes
0answers
14 views

Should outliers be remove first before identifying influential observations?

I have constructed a logistic regression model. I used half-normal probability plot and detected two outliers, which I removed. Then I want to identify influential observation, in order to improve the ...
0
votes
0answers
11 views

Adjusting for outlier in Fractional logit in R when dv is very small proportion

I used the code from this site: http://stackoverflow.com/questions/19893133/fractional-logit-model-r to estimate a fractional logit model. There are 90 observations in my dataMy dependent variable is ...
1
vote
1answer
47 views

Cook's D, testing for outliers

I am working on a multiple linear regression and I want to check for outliers using Cook's D. I have a problem interpreting it, as there are many points above the 4/N line, but only one is >1. How ...
0
votes
1answer
42 views

Eliminating observations with big residuals in regression

I busy with a regression model that seems to have heteroscedasticity. The model has 6 independent variables and one dependent variable. I did the regression and noticed heteroscedasticity. I then ...
1
vote
1answer
80 views

Choosing a k-value for Local Outlier Factor (LOF) detection analysis

I have a set of three-dimensional data, and I'm trying to use Local Outlier Factor analysis to identify the most unique or strange values. How does one decide the k-value to use in LOF analysis? I ...
1
vote
1answer
67 views

Issues in auto.arima algorithm when using external regressors and outlier correction

auto.arima is an automatic arima modeling function in forecast package in R that uses information criterion(example: AIC/BIC) to ...
2
votes
0answers
29 views

Repeated sampling with outliers

I have a situation where I'm sampling a value (computer program runtimes, not that that should make a difference) where the sampled values are presumed to be distributed around an unknown "true" value ...
2
votes
1answer
35 views

Outliers and chi-squared goodness of fit

Do outliers lead chi-squared tests of goodness of fit to fail?
0
votes
0answers
24 views

Determining the p-value of two consecutive residuals

I am performing an outlier detection test in a monthly process to detect errors (for each month I have more or less 22 business days). I am using a Simple Linear Regression model. What would be the ...
0
votes
0answers
19 views

Finding outliers across experiments

I have data across six different biological experiments (of varying pH levels) for around 150 different cases (but not all cases are found in each experiment), looking like this: ...
0
votes
0answers
32 views

How can I determine if a single data point comes from a distribution of points in 2 dimensions?

I've got a distribution of points in xy space and I would like to determine if a new data point belongs to the existing distribution. Each red point represents a species' size and its home range or ...
0
votes
0answers
31 views

Finding a confidence interval for observations being outliers

Suppose I have a sample with sample size $N$ that is obtained experimentally, e.g. I have counted the number of birds at a certain location at a certain time. Now suppose that the sample (the number ...
0
votes
1answer
66 views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
1
vote
0answers
79 views

Multivariate Outlier Detection with Robust Mahalanobis

I am searching some documents and examples related multivariate outlier detection with robust (minimum covariance estimation) mahalanobis distance. I have 6 variables and want to plot them to show ...
0
votes
1answer
25 views

What is the goodness of fit for Robust regression? Or how can we validate the model?

I have a data which is Non-Normal and has some outliers. I tried fitting Linear regression after removing outliers and robust regression without removing outliers. I wish to check which would be ...
1
vote
0answers
46 views

Unsupervised learning outlier detection

I have a dataset that looks as follows userid⇥week1⇥week2 ⇥week3⇥week4⇥week5⇥week6⇥week7 ...
0
votes
0answers
20 views

Multivariate Mahalanobis Distance Vector Normalization

I have a vector including different variables with different scales. For instance ''a'' presents ''dollar value'' in billions.''b'' is a ratio, presents value divided by quantity and it ranges 0 to 1 ...
5
votes
0answers
50 views

Problems with Outlier Detection

In a blog post Andrew Gelman writes: Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by ...
0
votes
0answers
23 views

References on analysis of outliers

I need some advice on literature, preferably papers, that explain the basics of outlier analysis. Does anyone have any tips?
1
vote
3answers
267 views

Ask for suggestions on clustering methods on a large dataset with mixed types of variables

I need to build segmentation on a large customer dataset with more than 300K records and many variables, including continuous like income and age, ordinal like education level and membership level, ...
1
vote
1answer
52 views

Popular methods for outlier detection (right skewed distribution)

What are the popular methods for outlier detection in univariate data, which do not assume normal distribution?
1
vote
1answer
117 views

Outlier detection with data (which has categorical and numeric variables) with R

Scenario I have a project about fraud detection where i need to find outliers by kmeans. I have a dataset about bank credits length of 1000. There are 21 columns (14 categorical, 7 numeric ...
0
votes
0answers
28 views

what is name outlier detection method

I want to find the name of method for outlier detection. this method calculate standard deviation and mean of data set. sample that is not in (mean-sd mean+sd) is an outlier.
3
votes
1answer
53 views

Does MCD estimator suffers from swamping effect?

If there are multiple outliers in the data set, the Mahalanobis distance suffers from masking and swamping effects. In order to rectify this problem, robust estimation of location and scale, such as ...
8
votes
2answers
489 views

Removing outliers from data - maximum number of outliers that you can remove?

I have a couple of outliers in my data and I was wanting to exclude them to see if this changes the results. In you opinion, what is the maximum number of outliers one should restrict themselves to? ...
1
vote
0answers
14 views

Running/Online Outlier Detection?

I get a stream of values online from a potentially non-stationary distribution, and would like to calculate whether a value is "rare" i.e. higher values only occur once in every 100 or 1000 values. It ...
0
votes
0answers
29 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...
0
votes
0answers
21 views

Statistical test to assess that a bunch of point diverge from background distribution

I've to set of points. First a set of points representing the background, that are very close to the identity line (slope=1). And a second set of points that seems to diverge from this identity line ...