An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
22 views

Outliers in Boxplots when calculating means

I have a dataset for which I am making boxplots. I do not want to include the outliers in the box plot so I give an argument outline=FALSE in the command. In the next step I want to put the mean ...
0
votes
0answers
16 views

Removing outliers in R plot function [migrated]

I'm trying to do a scatterplot in R and would love to remove an outlier I've already identified. My plot function: ...
0
votes
0answers
16 views

Identify outliers with median-absolute-deviation for timeseries data

I am having trouble understanding this particular method of detecting outliers in a time series. Below is the problem: I have a region-of-interest containing 15 voxels. Each voxel contains values ...
3
votes
4answers
199 views

Confused by location of fences in box-whisker plots

In one type of box-whisker plot, the fences at the ends of the whiskers are meant to indicate cutoff values beyond which any point would be considered an outlier. The standard definitions I've found ...
0
votes
1answer
14 views

Outliers and Influential observations in fixed effects regression

I am running a fixed effects regression with a very unbalanced panel data. There are a lot of residuals. Like for half of my observations I get large residuals. So I do not want to simply remove them ...
1
vote
0answers
21 views

Unbalanced Panel data using R - Removing outliers and heteroskedastcity

I am new in R and it’s my first time using it so I’ll appreciate the help. I am estimating income elasticity for electricity consumption using budget shares. I have data for 8 regions categorized into ...
0
votes
0answers
4 views

Removing MULTIPLE outliers in regression model in R [migrated]

this is in R ok so i've used cook distances to identify the points i would like to remove from a dataset of 506 variables that i have. i am able to remove ONE point (number 369) as follows: ...
1
vote
0answers
24 views

Identifying outliers from binned data

I have binned data (x-axis) that I've plotted against frequency (y-axis) to see the distribution of the bins and I got this scatterplot:- The bins are arranged from the lowest to the highest. As you ...
5
votes
2answers
61 views
+100

Back-testing or cross-validating when the model-building process was interactive

I have some predictive models whose performance I would like to back-test (i.e., take my dataset, "rewind" it to a previous point in time, and see how the model would have performed prospectively). ...
3
votes
0answers
57 views

How are outliers dealt with in R after detected? [closed]

Once outliers in time series are detected in R how exactly are they dealt with before forecasting? I dont want commands to use i would like the method. Please do not give any answers to do with ...
2
votes
0answers
38 views

what to do with ridiculous but valid leverage points

So I'm having some difficulty fitting a linear model to the data (see other post here glm model fit - can't find a family/link combination that produces good fit). In particular, I'm worried ...
0
votes
0answers
9 views

remove factor level if factor level has outlier in any of other columns in R dataframe

Hello I have a dataframe with one column a factor of patient id's, other columns of continous variables. I want to remove patients from the dataframe if they have an outlier observation in any of the ...
2
votes
3answers
222 views

Putting less weight on certain data points in a series for forecasting

I have a data set that contains outliers (big orders) i need to forecast this series taking the outliers into consideration. I already know what the top 11 big orders are so i dont need to detect them ...
1
vote
0answers
13 views

Is E-Divisive with Medians (the Twitter BreakoutDetection algo) robust and efficient?

There are quite a few algorithms to detect changepoints, outliers, mean shifts, trend shifts etc. out there. Recently I've stumbled upon BreakoutDetection and while it's new and shiny I'd like to know ...
0
votes
2answers
42 views

How to show that a dataset does not contain significant outliers?

I have largish dataset: there is 200 variables and 100 samples. How could I show that the dataset does not contain any significant outliers? All variables have the same unit (millimeters) and have ...
1
vote
0answers
20 views

before clusterisation, should I remove observations with too few measurements?

I have a very unevenly distributed dataset of 462 twitter users. During the window of observation, some of these users have produced as many as 2000 tweets, while others as few as one. My end is to ...
0
votes
0answers
19 views

Fat-tailed data and SVM

Does SVM perform poorly when fat-tailed data with outliers is used? What are some things that could be done to improve learning with such data? Does the choice of kernel and/or kernel parameter ...
0
votes
0answers
23 views

Correlation analysis while detecting outliers

I have simple dataset here. Supposed I want to find out which customers who bought a certain item are more likely to come back after 10 months. I have 2 sets of data The repeat purchase % of users ...
3
votes
0answers
16 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
4
votes
1answer
72 views

Intelligently selecting outliers

I'm trying to remove what might be considered "unreasonable" data by evaluating the percent error in the mean and square root of the variance. Here's the setup: Let's say I have three bids on a ...
0
votes
1answer
24 views

Spike detection and removal in position data

Is there any good filter to remove big spikes in position data? I think lowpass filter should be good but is it possible to filter 2D position data with assumption its joint distributed? I mean, not ...
1
vote
0answers
9 views

how to determine if a test run in only one geographic area leads to activity different than in other geographic areas

I was recently given data from an experiment that I did not design. In this study, the people running it changed behavior in 1 designative market area (DMA) but not the other DMAs (209 other DMAs). ...
0
votes
0answers
26 views

What do you do with outliers when developing statistical models?

I am a beginner so I have an extremely tough time dealing with outliers. I wanted to ask the community to help me understand rule of thumbs or anythng that would help me deal with these questions ...
0
votes
0answers
18 views

Anomaly Detection with very small number of positives [closed]

I am trying to detect anomalies in a population comprising of 10 features and around 90,000 observations. Past investigations have revealed 18 positives. Given limited data for supervised learning, I ...
0
votes
0answers
44 views

Outside 1.5 times inter quartile range yet not outlier

I have a data set with the following density histogram and box plot. Summary info as given R function. ...
2
votes
1answer
128 views

how to determine skewness from histogram with outliers?

I have the following histogram created in Minitab. I am wondering whether this histogram is actually positively skewed, negatively skewed, or symmetric. By observing the graph itself, it seems that ...
2
votes
1answer
63 views

Elastic net: dealing with wide data with outliers

Recently I was working on a dataset with ~300 observations and 1500 predictors. I used the glmnet package in R to fit an elastic net model, which gave me a ...
6
votes
4answers
145 views

How to prepare/construct features for anomaly detection (network security data)

My goal is to analyse network logs (e.g., Apache, syslog, Active Directory security audit and so on) using clustering / anomaly detection for intrusion detection purposes. From the logs I have a lot ...
0
votes
1answer
37 views

How to predict in advance that a smart meter is failing?

I have an electricity consumption data set collected by smart meters over a year and a half for every hour. The objective is to predict whether the meter could fail earlier than it actually fails.I ...
6
votes
2answers
136 views

What is a good method to identify outliers in exam data?

I give my students an exam that has 8 questions on it. Each question is about a particular topic. The exam is made up on the fly by randomly selecting 1 question for each topic from a pool of ...
2
votes
0answers
58 views

Basic methods for detecting outliers

Let $X$ be a matrix of $n$ rows and $m$ columns. $n$ is the number of samples and $m$ is the number of gene expressions. Gene expressions are basically numerical continuous values. Assume we have a ...
1
vote
1answer
56 views

Detecting anomalies in a time series where new data points will be continuously added

I have a time series data and I will be adding more data points in a consistent manner. I want to figure out whether the new data point added is an outlier, in regards to the previously observed data ...
0
votes
0answers
14 views

Should outliers be remove first before identifying influential observations?

I have constructed a logistic regression model. I used half-normal probability plot and detected two outliers, which I removed. Then I want to identify influential observation, in order to improve the ...
0
votes
0answers
11 views

Adjusting for outlier in Fractional logit in R when dv is very small proportion

I used the code from this site: http://stackoverflow.com/questions/19893133/fractional-logit-model-r to estimate a fractional logit model. There are 90 observations in my dataMy dependent variable is ...
1
vote
1answer
50 views

Cook's D, testing for outliers

I am working on a multiple linear regression and I want to check for outliers using Cook's D. I have a problem interpreting it, as there are many points above the 4/N line, but only one is >1. How ...
0
votes
1answer
44 views

Eliminating observations with big residuals in regression

I busy with a regression model that seems to have heteroscedasticity. The model has 6 independent variables and one dependent variable. I did the regression and noticed heteroscedasticity. I then ...
1
vote
1answer
101 views

Choosing a k-value for Local Outlier Factor (LOF) detection analysis

I have a set of three-dimensional data, and I'm trying to use Local Outlier Factor analysis to identify the most unique or strange values. How does one decide the k-value to use in LOF analysis? I ...
1
vote
1answer
78 views

Issues in auto.arima algorithm when using external regressors and outlier correction

auto.arima is an automatic arima modeling function in forecast package in R that uses information criterion(example: AIC/BIC) to ...
2
votes
0answers
29 views

Repeated sampling with outliers

I have a situation where I'm sampling a value (computer program runtimes, not that that should make a difference) where the sampled values are presumed to be distributed around an unknown "true" value ...
2
votes
1answer
37 views

Outliers and chi-squared goodness of fit

Do outliers lead chi-squared tests of goodness of fit to fail?
0
votes
0answers
27 views

Determining the p-value of two consecutive residuals

I am performing an outlier detection test in a monthly process to detect errors (for each month I have more or less 22 business days). I am using a Simple Linear Regression model. What would be the ...
0
votes
0answers
19 views

Finding outliers across experiments

I have data across six different biological experiments (of varying pH levels) for around 150 different cases (but not all cases are found in each experiment), looking like this: ...
0
votes
0answers
37 views

How can I determine if a single data point comes from a distribution of points in 2 dimensions?

I've got a distribution of points in xy space and I would like to determine if a new data point belongs to the existing distribution. Each red point represents a species' size and its home range or ...
0
votes
0answers
35 views

Finding a confidence interval for observations being outliers

Suppose I have a sample with sample size $N$ that is obtained experimentally, e.g. I have counted the number of birds at a certain location at a certain time. Now suppose that the sample (the number ...
0
votes
1answer
68 views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
1
vote
0answers
82 views

Multivariate Outlier Detection with Robust Mahalanobis

I am searching some documents and examples related multivariate outlier detection with robust (minimum covariance estimation) mahalanobis distance. I have 6 variables and want to plot them to show ...
0
votes
1answer
28 views

What is the goodness of fit for Robust regression? Or how can we validate the model?

I have a data which is Non-Normal and has some outliers. I tried fitting Linear regression after removing outliers and robust regression without removing outliers. I wish to check which would be ...
1
vote
0answers
47 views

Unsupervised learning outlier detection

I have a dataset that looks as follows userid⇥week1⇥week2 ⇥week3⇥week4⇥week5⇥week6⇥week7 ...
0
votes
0answers
21 views

Multivariate Mahalanobis Distance Vector Normalization

I have a vector including different variables with different scales. For instance ''a'' presents ''dollar value'' in billions.''b'' is a ratio, presents value divided by quantity and it ranges 0 to 1 ...
5
votes
0answers
51 views

Problems with Outlier Detection

In a blog post Andrew Gelman writes: Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by ...