An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

learn more… | top users | synonyms (1)

0
votes
0answers
8 views

detecting outlier of very small data set with assigned error

I have this (very small) data set: -0.032 +/- 0.011 0.020 +/- 0.011 0.025 +/- 0.010 I haven't see any test including the error on the data. In addition, the value of -0.032 can be rejected by a Grubbs ...
1
vote
0answers
10 views

how to remove outliers prior to multiple imputation

A colleague came to me with the following problem. She has a complex, multivariate data set, in which respondents completed a number of measures with anywhere from 6 to 30 Likert type items for each ...
0
votes
0answers
13 views

Finding proper cut off for filtering?

I had a set of data (149876 rows and 1200 columns)and I have found outliers points for each row. Now I want to determine a proper threshold for filtering my outliers. The distribution of my data is ...
0
votes
0answers
12 views

Modeling farm socio/demographics to determine influence on net income

I am attempting to isolate the effect on net income of incorporating a certain type of marketing strategy for farms. My initial OLS model produced positive and expected results, but when I graphed my ...
0
votes
0answers
20 views

Outlier detection for different sensor types

I have several sensor type: Sound (Range: 0 - 20) Light (Range: 0 - 600) Temperature Both sound and light sensors have an expected range for their values. Temperature sensors are expected to stay ...
0
votes
0answers
30 views

Sensor fault detection algorithm explanation

I am attempting to implement a sensor fault detection algorithm from a white paper I found here: http://www.hindawi.com/journals/mpe/2013/712028/ref/ You should not have to read the article to ...
0
votes
0answers
18 views

alternatives for OC-SVM for one class unsupervised classification in R

I have been working with OC-SVM for one class unsupervised classification using R. I would like to try other techniques for the same purpose. However, I find lots of outlier detection techniques, like ...
0
votes
1answer
23 views

Determining window sizes of varying length sub-sequences of time series data for outlier/discord detection

I'm working on some outlier detection methods for seasonal time series data. Basically I want to automate discord detection, i.e. suppose the time series could be split into multiple windows such that ...
1
vote
0answers
18 views

How to handle outliers in Poisson regression?

Consider the following count data: df <- data.frame( count=c(0,1,2,3,4,5,9,20), freq=c(1120,42,10,5,1,1,1,1) ) I want to use a quasi-poisson regression ...
0
votes
1answer
43 views

Outlier filtering in 2D data in python

I have following data given: My curve fits it acceptable for my needs. I use here 4th degree polynomial. (data is limited to 0-100 percent range for both axis!) What I want to try now is to filter ...
0
votes
0answers
20 views

DBSCAN application for detection of anomalous instants in a particular time series

I have time series/matrix with cpu utilization of 4 servers (about 17k points). I am trying to use DBSCAN algorithm to find out which server is operating suspiciously compared to other using the ...
0
votes
0answers
15 views

Use deep belief networks for unsupervised anomaly detection

I am working on anomaly detection on data with a large number of variables (>50) with continuous values. As I have read that deep belief networks can be used for unsupervised anomaly detection on ...
0
votes
0answers
44 views

find outlier in 1D array

I have a satellite images, which have none values (which are not all equal to 0) on right and left side of an image and it is not a strait line. I would liked to write a program, which finds the ...
0
votes
0answers
33 views

How to detect outliers?

I have a matrix where the rows are the data points (samples) and the columns are the features (predictors). Let's say I have 1000 data points and 20 features, i.e. the matrix is of size 1000 x 20. ...
0
votes
0answers
15 views

Instrumental variable changes significance of other regressors

I have a linear regression:I am regressing price index and price volatility on some number of conflict events (dependent variable). Looking for a causal relationship between price index and number of ...
1
vote
0answers
23 views

Residuals of logistic regression and detecting outlying observations

Is it correct to say that the residuals of logistic regression in practice have a bimodal distribution? And how would this make it harder to detect outlying observation?
3
votes
4answers
106 views

Difference between Anomaly and Outlier

Please let me know what is the difference between Outlier and Anomaly in the context of machine learning. My understanding is that both of them refer to the same thing.
1
vote
0answers
17 views

Correction for non-normal residuals where residuals are not correlated or show heteroskedasticity

I know the implication of non-normal residuals is uncertain statistical tests because the SEs are inefficient. Can I apply the Newey-West to calculate standardized Standard Errors for an OLS ...
3
votes
1answer
65 views

Detect outliers (anomalies) in salary data

I have a dataset with over 10,000 employees and I'm interested in performing some tests to identify salary anomalies. For example, paychecks for some employees might look like this: ...
6
votes
2answers
65 views

R - Approach to find outliers/artefacts in blood pressure curve

Do you guys have an idea how to approach the problem of finding artefacts/outliers in a blood pressure curve? My goal is to write a program, that finds out the start and end of each artefact. Here are ...
1
vote
0answers
35 views

Identify outlier in a group with a test statistic

I am trying to identify the outliers from a group for a research article. I wanted to know if there was a formal statistical method of doing so. Here is an example of the data: There are 500 students ...
16
votes
3answers
419 views

How are Random Forests not sensitive to outliers?

I've read in a few sources, including this one, that Random Forests are not sensitive to outliers (in the way that Logistic Regression and other ML methods are, for example). However, two pieces of ...
1
vote
1answer
23 views

mvBacon from R to Python [closed]

I use package "robustX" in R in order to compute the Mahalanobis distance in a dataset to detect outliers, using more than one variable. Specifically the code I am using is: ...
0
votes
0answers
13 views

Diagnostic measure not influenced by transformations of response variable

What are the diagnostic measures (like Cook's distance, H matrix, DFFITS, DFBETAS) that are not affected by the transformation of the response/dependent variable? And why?
0
votes
0answers
74 views

When the Median Absolute Deviation (MAD) is zero

Suppose my data look like the following: (10, 10, 10, 10, 10, 0) Would it be possible to remove an outlier in this distribution using the median absolute deviation? Of course, you wouldn't need to ...
0
votes
1answer
63 views

Find outliers in time series with unknown distributions

I want to detect outliers in time series data like the two outliers in the image below. At first I tried LOF, which didn't work well and outlier detection methods based on normal distributions ...
0
votes
1answer
21 views

Reducing density of outliers

I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days) Now the TTB values less than 14 or so are possible. But not ...
0
votes
1answer
36 views

Cook's D and T-Tests

Why is it more important to conduct a cook's d in regression than when doing a t-test to compare means of two groups? I know it has something to do with the outliers not having much leverage but why ...
1
vote
0answers
24 views

Creating Anomaly Detection algorithm

I'm Interested in the time series anomaly detection for security log files , on the following link I found similar case related to my interest: Time Series Anomaly Detection with Python , at the end ...
0
votes
0answers
37 views

How to Identify *Sequences of Outliers (all 0's)* in Periodic Time Series Data (unsupervised learning)?

I have a large time series dataset of nearly 2000 email servers. Data is hourly pair-wise number of emails sent between each pair of servers over one year. The data is very periodic, with a strong ...
1
vote
2answers
68 views

Detect if an incoming value in streaming data is an outlier

I am reading a sensor that gives data. Sometimes some data is false. I can store some samples before and I would like to detect a glitch on the fly. Process : Values are integers (distances in ...
1
vote
1answer
67 views

how to find outliers from high-dimensional data set?

The data has about 40 features and 500,000 instances. And the data is sparse. I wish to fit a svm model with the data. To fit svm, I need to first scale the data. However, if the data contains many ...
0
votes
0answers
25 views

Logistic Regression - Multivariate Outliers

I am running a Logistic Regression and checking for the assumption of Multivariate Outliers. My Mahalanobis and Cook's distances are all within the acceptable values, however some of the Leverage ...
4
votes
0answers
77 views

Detecting outliers in non-normal distribution data

I'm working with data from a resistivity test. However, during the test it is common that a few measurement points are wrong due to technical failure. So I want to find and remove these points. I ...
0
votes
0answers
41 views

Removing outliers from asymmetrical, multivariate, categorical data?

I'm doing a project which asks the question of whether common ivy (Hedera helix) is more abundant on some species of tree than others. I did several transects and looked at each tree within the ...
1
vote
0answers
41 views

How to check the number of outliers in my data?

I have multivariate data and want to check for outliers. I guess there are both outliers in the predictor variables and the dependent variable. My idea was to fit robust regression methods after ...
0
votes
0answers
11 views

Analysing data obtained from self-report inventries

I am currently trying to analyse the data obtained from self-report inventories. The sample size is 67 with two groups of 44 and 23 subjects. Moreover, there exist three time points (pre-, post- and 3 ...
0
votes
1answer
34 views

Outlier detection for a univariate categorical variable?

Does anyone know an outlier detection method for a univariate categorical (nominal, unordered) statistical variable? Without any assumptions about the categorical variable distribution (non-parametric ...
0
votes
0answers
7 views

ANOVA with wide range signals

I have some signals from which I would like to detect outliers. I have thousands of columns. And I would like to apply ANOVA on those signals. The problem is that, the range are too wide. I have ...
1
vote
0answers
20 views

Problems in finding outliers and leverage points in non-linear regression

I'm implementing diagnostic of non-linear regression model $(y=ax^b)$. I'm trying to find out where outliers and leverage points in my model ...
0
votes
0answers
24 views

Proper approach to Gamma-distributed data prediction with measurement errors in outliers

My task is to predict Gamma-distributed data with a large number of extreme-valued outliers caused by measurement error (i.e. the machine that records the values intermittently malfunctions). My ...
0
votes
1answer
44 views

using MARS regression on data with outliers - R

In some previous asked questions, I was told to not delete the outliers, because they contain valuable information. After testing different regression, I came to the conclusion that until now, the ...
2
votes
2answers
109 views

Cross-Validation driven outlier removal

Can I turn the outlier removal problem into a model-selection problem and use cross-validation to solve it? Take a situation similar to this question: you have a mixed model you want to fit against ...
0
votes
0answers
12 views

Finding Anomalies in skewed data

I have the following highly skewed variables Each data point represents an observation and I'm trying to find observations that are not like the others. For each variable, I tried finding outliers ...
1
vote
0answers
76 views

Detection of outliers in 3 dimensions

I have a data set that has x,y,z variables. z is sampled data based on location (x,y). How can I detect potential outliers in z-value, thus corresponding to either, a particular (x,y) location, or a ...
0
votes
0answers
84 views

remove outliers before doing regression

I'm having troubles eliminating the outliers out of my data, before running a regression through it. data set structure: ...
0
votes
1answer
33 views

How to Exclude Lower Order Lags from a Custom ARIMA Model in SPSS Forecasting Module

I'm trying to use an ARIMA model to generate forecasts using monthly revenue data that exhibits both trend and seasonality. I've used the Expert Modeler from the SPSS Forecasting Module to determine ...
0
votes
0answers
13 views

What is more time efficient (for a given level of accuracy) - ensemble several smaller SVMs or train one large one?

I am building a "one class svm" to help detect outliers. I have rather large dataset of 3.2M points in $\mathbb{R}^4$ that I'd like to apply one class SVMs to. Now, I've read that SVMs have a fitting ...
0
votes
0answers
11 views

dealing with outliers in neural networks training

I have an idea for training robust to outliers. Do a batch gradient descent, save the weight changes for every sample but don't use one that are too different from the others from a same class. Would ...
1
vote
0answers
24 views

Negative value when using the Outlier Labeling Rule

I think I have some outlier values in my data set so I'm using the Outlier Labeling Rule (Hoaglin, D. C., Iglewicz, B., and Tukey, J. W. [1986]. Performance of some resistant rules for outlier ...