An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

learn more… | top users | synonyms (1)

1
vote
1answer
27 views

What comes first: outlier detection or model selection?

I'm fitting a GLMM (mixed logistic regression) in R. I have five covariates. For model selection, I'm using glmmLasso() (in R) to determine which of the five covariates and their interactions should ...
0
votes
1answer
18 views

Finding outliers in multiple dimensions

I'm working on dataset which isn't normally distributed. It contains three dimensions: cost, discount and profit. I'm trying to find outliers in all these dimensions. I used $\text{z-score}$ to find ...
0
votes
1answer
11 views

picking out outliers from a GLM in R

I recently fitted a beta distributed GLM using R (and the betareg package). as you can see the model is a reasonably decent fit, however there are a few outliers. i would like to run the model again ...
0
votes
0answers
25 views

Why leverage measure the distance of the ith observation from the center of the x space? [duplicate]

I know the definition of leverage points in regression, that is $h_{ii}=x_{i}'(X'X)^{-1}x_{i}. $ In many places and text books, they always say that leverage is a standardized measure of the distance ...
3
votes
0answers
13 views

How to detect a relatively small level shift(leakage) in an hourly water flux time series in an area?

Background I'm working on a project which aims to use the history data about a water flux to detect whether there is a leakage happened. The data is hourly collected and among about 4 months. I've ...
0
votes
0answers
22 views

Why different diagnostic tests for detecting outliers in linear regression don't agree with each other?

I have one predictor and one variable, which means that the "cutoff" Mahalanobis value is > 3.84 at 0.05 significance level; covariance ratio should be between 0.88 and 1.12 (1+-3x(1+1)/50); the ...
1
vote
0answers
5 views

Are there kernel-based one-class sparse kernel-based outlier detection methods, e.g. one-class Relevance Vector Machine?

I have a commercial outlier detection problem in moderate dimension (8-25). We have a limited number of true positive tags and can roughly evaluate performance of various methods. So far, the ...
0
votes
0answers
5 views

Intuitive explanation of Grubb's test

Can anyone give an explanation of Grubb's test for outliers? I've found many resources giving steps to calculate it, but none even attempting to talk about why that equation should be a good outlier ...
0
votes
0answers
20 views

How to deal with outliers and feature selection simultaneously?

I've been given some data and need to pick what I consider to be the best features from it and use them to build models that fit the data. My issue is that all the tests I've seen for outliers assume ...
0
votes
1answer
37 views
+50

anomaly/outliers detection in a multilabel dataset on the outcomes

Assuming a multilabel dataset contains a few wrong data. If so, is there a way to predict those wrong outcome given the fact there is a 'pattern' in the predictors? Let's use 'baby and silly' ...
0
votes
0answers
25 views

Outliers detections in time-series

I am searching algorithms for detecting outliers in a time-series data. I see that there are a lot of algorithms and they have an implementation in R. But i don't find any explanation on how they ...
0
votes
0answers
12 views

Which Statistical model should I Choose to fit the Data?

I am struggling with this problem for quite a few days.So far, I have used simple Box-Plot method to pick out the outliers for each location and Diseases. And how can I get outliers after fitting any ...
2
votes
0answers
7 views

Check for outlier in random effects

I have 3 treatements and each treatment is given to 2 random people from which 2 samples is taken from each person that gets the specific treatment. The dataframe in R below shows a sample dataset ...
0
votes
0answers
32 views

Testing for discords in seasonal time series data

I'm trying to find a way to detect discords in seasonal data. I have an algorithm that can select the most likely sub-sequence to be a discord, but what I'm missing is an actual test. I know that ...
1
vote
0answers
30 views

Outlier transformation or no transformation?

I have two group's of 25 people each. I want to know if there a difference between them on a past experience of happiness . I have a test with 100 question in it and all question have an a and b ...
1
vote
0answers
32 views

Linear Regression – When is Bad Data “Too Much”?

I am doing a multiple linear regression analysis. One variable, which I think may be quite predictive, has known bad data. I am currently sampling and using analysts to independently verify the ...
5
votes
1answer
38 views

Cook's distance in detecting outliers

According to my understanding, Cook's distance measures the influence of each observation by excluding points when fitting a model. So I assume it could be an reasonable approach for outlier ...
0
votes
0answers
14 views

Outliers detection or some roubust metrics on long tail sqewed distributions

I have a distribution of user sessions on the web site in the following format date,sessionId,price 2010-01-01,1,0 2010-01-01,2,0 2010-01-01,3,10 ... And I am ...
0
votes
0answers
18 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
24 views

How is finding outliers using extreme value theory different from setting threshold on a pdf of normal events?

Suppose we assume that the data follows a Gaussian distribution. We can set a threshold on its pdf to find outliers. How would it be different from setting a threshold on a pdf, fit the exceedances ...
1
vote
1answer
125 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
1
vote
0answers
11 views

Get rid of values that pollute (or obfuscate) aggregate statistics

I consider myself a beginner in statistical analysis, and I want some advice on a problem that I have encountered in a data analysis task. I am monitoring a system's latency for producing results, ...
2
votes
0answers
54 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
1
vote
0answers
38 views

Use linear regression to detect outliers and leverage points

I want to use linear regression to pre-process the data (e.g find outliers) so that I can use techniques like ANOVA to analyze the data. The goal is not to fit a regression model. I saw two posts ...
1
vote
1answer
70 views

XGBoost (Extreme Gradient Boosting) or Elastic Net More Robust to Outliers

I have recently been doing work with predictive models for a continuous response. I am doing a comparison between Elastic Net (glmnet) package in R and XGBoost ...
3
votes
0answers
27 views

Outlier removal for univariate and multivariate analysis

I have a biological data set on which I would like to do both univariate and multivariate analysis, and try to find correlation of features to a response. Should I remove univariate outliers and do ...
1
vote
0answers
23 views

Stationary time series with outliers

Does anyone have stationary time series data with some outliers from real life? I'd like to try my robust estimation method. Thank you!
2
votes
0answers
24 views

How should I interpret/follow-up on mixed logistic regression (GLMM) diagnostics?

I have experimental data (n subjects = 64) in which the response variable, accuracy (0 or 1), was measured 9 times within subjects. My predictor is Condition (A vs B) measured between subjects. I ...
0
votes
0answers
23 views

Detecting the first outlier in a series of inter-call times

Question: how can I detect the first outlier in a sequence of inter-calls times? Imagine I have a sequence of telephone calls and that the time from call $N$ to call $N-1$ is: lags = {1.1, 1.2, ...
22
votes
6answers
2k views

Is it OK to remove outliers from data?

I looked for a way to remove outliers from a dataset and I found this question. In some of the comments and answers to this question, however, people mentioned that it is bad practice to remove ...
4
votes
3answers
207 views

Sensitivity of the mean to outliers

Is the mean sensitive to the presence of outliers? I initially thought it wasn't, because a small amount of observations shouldn't have much impact, but was told that since those observations have ...
0
votes
0answers
10 views

Method for Outlier Identification in High Dimensions [duplicate]

Is there "state of the art" methods for outlier identification in high dimensions? I have come across PCOut algorithm and a survey paper "A survey on unsupervised outlier detection in ...
0
votes
0answers
10 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
2answers
40 views

Removing outliers from dataset [duplicate]

I am conducting separate regression models for two of my hypotheses. If I spot outliers in one case of conducting the regression model can I then bring them back for my second? Or do these outliers ...
0
votes
0answers
41 views

Hidden Markov Model for anomaly detection

In Hidden Markov Model, it is possible to compute probability of observation sequence by applying forward algorithm given learned model. We can detect anomaly sequences by this algorithm simply by ...
0
votes
0answers
30 views

Regression: using confidence interval to remove outliers

I have to calculate a regression (for instrument calibration purposes) between Log(x) and y, where x is an environmental parameter, known to be log-distributed. Accurate measurement of x is difficult ...
0
votes
1answer
15 views

Can I use the Cook´s Distance to find outliers in a GAM?

Can I use the Cook´s Distance to find outliers in a GAM with binomial family? I have seen that is often used to find outliers in GLM but I did not find an application example for GAM. I therefore ...
0
votes
1answer
23 views

Exclude outliers when most scores are 0

I'm doing a word learning experiment for which I prefer my participants to have no prior knowledge of the words. Prior knowledge is determined by a pre-test. Most participants indeed know 0 words ...
0
votes
0answers
49 views

Outlier detection in multivariate data

I have a table with thirty variables and am interested in finding the outliers (rows). Assuming that my variables are independent I am hesitant whether I should use Cook's distance as it requires to ...
0
votes
1answer
30 views

Detect outliers in multi dimenstions

I tried to implement outlier detection for one dimension using inter quartile. for instance , a given variable cost or revenue or profit. but I'm missing outliers in other dimensions when running for ...
0
votes
0answers
52 views

outliers in Likert-sacele data

I have some data with Likert scale (1-5). In the boxplot, SPSS report some outliers (although the value is within the 1-5 range). As I read so far, removing outliers on Likert scale data is arguable. ...
0
votes
0answers
15 views

Deleting outliers in (almost) uniform graph

I wrote an algorithm that finds the centroids of rectangles which are on the same line and groups these rectangles together. I know that only those rectangles with a roughly equal size belong ...
3
votes
0answers
26 views

Rules of thumb for a proportion of outliers depending on the dimension

I am implementing and benchmarking different "robust" PCA (principal component analysis, see for instance Robust Principal Component Analysis?) for data that should align (I have no prior on the ...
0
votes
1answer
53 views

How to identify outliers and do model diagnostics for an lme4 model?

I need to identify outliers and high leverage points, and perform model diagnostics, in an lme4 model. For outliers and high leverage points, simply making a plot ...
0
votes
1answer
64 views

How to improve the performance of svm by using fuzzy membership function in R?

According to Lin and Wang, 2002, fuzzy support vector machine gives good performance on reducing the effects of outliers and significantly improves the classification accuracy and generalization. ...
1
vote
0answers
13 views

Should I use a Bayesian multilevel model to detect outliers?

I have approximately-log-normal price data. The data is hierarchically structured. Let's say there are three levels, so the notation for a log-transformed observation could be $x_{ijk}$, with the ...
3
votes
0answers
23 views

Detecting discontinuities in irregularly-spaced data

I am running physics simulations. Sometimes a simulation has insufficient resolution and produces discontinuities in the output variables. These are very easy to spot by eye in a scatterplot. I would ...
0
votes
1answer
59 views

What are some common approaches for estimating variability of right skewed, bimodal distributions?

I have a data set with the dollars in transactions a store has done historically from noon-2pm on Wednesdays. Most of the time, the store does no volume during this time (16 observations with the ...
0
votes
0answers
11 views

detecting outlier of very small data set with assigned error

I have this (very small) data set: -0.032 +/- 0.011 0.020 +/- 0.011 0.025 +/- 0.010 I haven't see any test including the error on the data. In addition, the value of -0.032 can be rejected by a Grubbs ...
1
vote
0answers
26 views

how to remove outliers prior to multiple imputation

A colleague came to me with the following problem. She has a complex, multivariate data set, in which respondents completed a number of measures with anywhere from 6 to 30 Likert type items for each ...