An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
2 views

how to determine if a test run in only one geographic area leads to activity different than in other geographic areas

I was recently given data from an experiment that I did not design. In this study, the people running it changed behavior in 1 designative market area (DMA) but not the other DMAs (209 other DMAs). ...
0
votes
0answers
15 views

Does THE discordance test for outliers exist? [on hold]

I have looked but can't find 'the discordance test for outliers'. Does this specific test even exist? And if it doesn't, which test would you suggest to measure discordance for outliers?
0
votes
0answers
25 views

What do you do with outliers when developing statistical models?

I am a beginner so I have an extremely tough time dealing with outliers. I wanted to ask the community to help me understand rule of thumbs or anythng that would help me deal with these questions ...
0
votes
0answers
13 views

Anomaly Detection with very small number of positives [closed]

I am trying to detect anomalies in a population comprising of 10 features and around 90,000 observations. Past investigations have revealed 18 positives. Given limited data for supervised learning, I ...
0
votes
0answers
34 views

Outside 1.5 times inter quartile range yet not outlier

I have a data set with the following density histogram and box plot. Summary info as given R function. ...
2
votes
1answer
60 views

how to determine skewness from histogram with outliers?

I have the following histogram created in Minitab. I am wondering whether this histogram is actually positively skewed, negatively skewed, or symmetric. By observing the graph itself, it seems that ...
2
votes
1answer
46 views

Elastic net: dealing with wide data with outliers

Recently I was working on a dataset with ~300 observations and 1500 predictors. I used the glmnet package in R to fit an elastic net model, which gave me a ...
6
votes
4answers
127 views

How to prepare/construct features for anomaly detection (network security data)

My goal is to analyse network logs (e.g., Apache, syslog, Active Directory security audit and so on) using clustering / anomaly detection for intrusion detection purposes. From the logs I have a lot ...
0
votes
1answer
33 views

How to predict in advance that a smart meter is failing?

I have an electricity consumption data set collected by smart meters over a year and a half for every hour. The objective is to predict whether the meter could fail earlier than it actually fails.I ...
6
votes
2answers
130 views

What is a good method to identify outliers in exam data?

I give my students an exam that has 8 questions on it. Each question is about a particular topic. The exam is made up on the fly by randomly selecting 1 question for each topic from a pool of ...
2
votes
0answers
52 views

Basic methods for detecting outliers

Let $X$ be a matrix of $n$ rows and $m$ columns. $n$ is the number of samples and $m$ is the number of gene expressions. Gene expressions are basically numerical continuous values. Assume we have a ...
1
vote
1answer
50 views

Detecting anomalies in a time series where new data points will be continuously added

I have a time series data and I will be adding more data points in a consistent manner. I want to figure out whether the new data point added is an outlier, in regards to the previously observed data ...
0
votes
0answers
12 views

Should outliers be remove first before identifying influential observations?

I have constructed a logistic regression model. I used half-normal probability plot and detected two outliers, which I removed. Then I want to identify influential observation, in order to improve the ...
0
votes
0answers
9 views

Adjusting for outlier in Fractional logit in R when dv is very small proportion

I used the code from this site: http://stackoverflow.com/questions/19893133/fractional-logit-model-r to estimate a fractional logit model. There are 90 observations in my dataMy dependent variable is ...
1
vote
1answer
40 views

Cook's D, testing for outliers

I am working on a multiple linear regression and I want to check for outliers using Cook's D. I have a problem interpreting it, as there are many points above the 4/N line, but only one is >1. How ...
0
votes
1answer
40 views

Eliminating observations with big residuals in regression

I busy with a regression model that seems to have heteroscedasticity. The model has 6 independent variables and one dependent variable. I did the regression and noticed heteroscedasticity. I then ...
1
vote
1answer
37 views

Choosing a k-value for Local Outlier Factor (LOF) detection analysis

I have a set of three-dimensional data, and I'm trying to use Local Outlier Factor analysis to identify the most unique or strange values. How does one decide the k-value to use in LOF analysis? I ...
1
vote
1answer
54 views

Issues in auto.arima algorithm when using external regressors and outlier correction

auto.arima is an automatic arima modeling function in forecast package in R that uses information criterion(example: AIC/BIC) to ...
2
votes
0answers
29 views

Repeated sampling with outliers

I have a situation where I'm sampling a value (computer program runtimes, not that that should make a difference) where the sampled values are presumed to be distributed around an unknown "true" value ...
2
votes
1answer
22 views

Outliers and chi-squared goodness of fit

Do outliers lead chi-squared tests of goodness of fit to fail?
0
votes
0answers
19 views

Determining the p-value of two consecutive residuals

I am performing an outlier detection test in a monthly process to detect errors (for each month I have more or less 22 business days). I am using a Simple Linear Regression model. What would be the ...
0
votes
0answers
18 views

Finding outliers across experiments

I have data across six different biological experiments (of varying pH levels) for around 150 different cases (but not all cases are found in each experiment), looking like this: ...
0
votes
0answers
26 views

How can I determine if a single data point comes from a distribution of points in 2 dimensions?

I've got a distribution of points in xy space and I would like to determine if a new data point belongs to the existing distribution. Each red point represents a species' size and its home range or ...
0
votes
0answers
28 views

Finding a confidence interval for observations being outliers

Suppose I have a sample with sample size $N$ that is obtained experimentally, e.g. I have counted the number of birds at a certain location at a certain time. Now suppose that the sample (the number ...
0
votes
1answer
55 views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
1
vote
0answers
70 views

Multivariate Outlier Detection with Robust Mahalanobis

I am searching some documents and examples related multivariate outlier detection with robust (minimum covariance estimation) mahalanobis distance. I have 6 variables and want to plot them to show ...
0
votes
1answer
21 views

What is the goodness of fit for Robust regression? Or how can we validate the model?

I have a data which is Non-Normal and has some outliers. I tried fitting Linear regression after removing outliers and robust regression without removing outliers. I wish to check which would be ...
1
vote
0answers
41 views

Unsupervised learning outlier detection

I have a dataset that looks as follows userid⇥week1⇥week2 ⇥week3⇥week4⇥week5⇥week6⇥week7 ...
0
votes
0answers
15 views

Multivariate Mahalanobis Distance Vector Normalization

I have a vector including different variables with different scales. For instance ''a'' presents ''dollar value'' in billions.''b'' is a ratio, presents value divided by quantity and it ranges 0 to 1 ...
5
votes
0answers
46 views

Problems with Outlier Detection

In a blog post Andrew Gelman writes: Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by ...
0
votes
0answers
21 views

References on analysis of outliers

I need some advice on literature, preferably papers, that explain the basics of outlier analysis. Does anyone have any tips?
1
vote
3answers
145 views

Ask for suggestions on clustering methods on a large dataset with mixed types of variables

I need to build segmentation on a large customer dataset with more than 300K records and many variables, including continuous like income and age, ordinal like education level and membership level, ...
1
vote
1answer
45 views

Popular methods for outlier detection (right skewed distribution)

What are the popular methods for outlier detection in univariate data, which do not assume normal distribution?
0
votes
1answer
78 views

Outlier detection with data (which has categorical and numeric variables) with R

Scenario I have a project about fraud detection where i need to find outliers by kmeans. I have a dataset about bank credits length of 1000. There are 21 columns (14 categorical, 7 numeric ...
0
votes
0answers
25 views

what is name outlier detection method

I want to find the name of method for outlier detection. this method calculate standard deviation and mean of data set. sample that is not in (mean-sd mean+sd) is an outlier.
3
votes
1answer
45 views

Does MCD estimator suffers from swamping effect?

If there are multiple outliers in the data set, the Mahalanobis distance suffers from masking and swamping effects. In order to rectify this problem, robust estimation of location and scale, such as ...
8
votes
2answers
456 views

Removing outliers from data - maximum number of outliers that you can remove?

I have a couple of outliers in my data and I was wanting to exclude them to see if this changes the results. In you opinion, what is the maximum number of outliers one should restrict themselves to? ...
1
vote
0answers
13 views

Running/Online Outlier Detection?

I get a stream of values online from a potentially non-stationary distribution, and would like to calculate whether a value is "rare" i.e. higher values only occur once in every 100 or 1000 values. It ...
0
votes
0answers
26 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...
0
votes
0answers
21 views

Statistical test to assess that a bunch of point diverge from background distribution

I've to set of points. First a set of points representing the background, that are very close to the identity line (slope=1). And a second set of points that seems to diverge from this identity line ...
3
votes
1answer
73 views

Is it possible to seed RANSAC with a given line?

I am analyzing a stream of data and I want to seed every new instance with the best guess output (line) of the previous, so as to eventually converge. Given that Scikit Learn - RANSAC is an iterative ...
1
vote
1answer
100 views

Anomaly detection: multivariate Gaussian distribution

I am trying to do anomaly detection on a heterogeneous dataset (There are unknown groups present in the dataset). I want to try multivariate Gaussian distribution based approach, but I was thinking of ...
0
votes
0answers
68 views

How to determine the lower- and upper- tail cutoff values efficiently?

I have a long vector (~1.000.000 entries) of integers from 1 to 2500 each of which expresses the number of occurrences of some sort of event for a certain user. The data can be illustrated as: ...
7
votes
2answers
339 views

Outlier Detection on skewed Distributions

Under a classical definition of an outlier as a data point outide the 1.5* IQR from the upper or lower quartile, there is an assumption of a non-skewed distribution. For skewed distributions ...
1
vote
0answers
31 views

Quantitative qualification of outliers [closed]

Is there any quantitative method to qualify outliers that would help distinguish junk outliers from information-rich ones? For example, if multiple outliers look alike, there is a greater chance that ...
0
votes
0answers
55 views

Outliers in panel data

I have a panel data with 4 waves. The variable of interest for me is hourly pay which increases on average from wave to wave. I want to drop the observations with hourly pay in the bottom and the top ...
2
votes
1answer
41 views

Relation between $R^2$ and $R^2_{(i)}$

Is there a relation between $R^2$ and $R^2_{(i)}$ (where $R^2_{(i)}$ is the $R^2$ of a regression without the point ith. For example if the ith point is an outlier) without having to recalculate all ...
0
votes
0answers
118 views

Novelty and Outlier Detection in Unsupervised Learning Style

Currently I am looking for some method to do novelty and outlier detection. I found some good example here using scikit-learn (Link1). However, it is based on supervised learning and I believe the ...
0
votes
1answer
41 views

Novelty and Outlier Detection for Multi-label Data

I met a problem of using novelty and outlier detection for my multi-label data. For example, I have got some training data that is not polluted by outliers. However, the training data are with ...
0
votes
0answers
24 views

How to find mobile app usage outliers?

I am doing some analysis on the usage data of my company's mobile app. However, the QA team will do some prod-test at any time, which generates enormous amount of usage data and also obscure the ...