An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

learn more… | top users | synonyms (1)

0
votes
0answers
29 views

which is the most sutible technique to detect outliers? [closed]

i know a technique to detect outliers: 1- make a model & calculate residual for each data point 2- delete the top 10% residuals from the data 3- fit the data again that's fine but this leads ...
0
votes
0answers
8 views

Finding outliers on LiDAR for forests

I'm working with LiDAR data in college and focusing on preprocessing. I'm still graduating and this is my first project. So, I used the library of c++, Point Cloud Library (PCL) to deal with the ...
2
votes
0answers
31 views

Dealing with noisy/mislabelled dataset

I have several datasets where each instance has numeric label assigned by a human that can take values between 1 and 5. After doing a manual inspection of one of these datasets, I noticed the ...
0
votes
1answer
23 views

Finding outlier values for non-normally distributed data

I have univariate data (38 is the sample size).The distribution is certainly not normal. How can I find the outliers? I used z-score but am not getting a desired result.
1
vote
0answers
16 views

Transformation and/or Winsorizing?

I want to compare two group of 24 and 28 people with t-test on type of activity (5 different's types of activity and a total), later one the same value will be use in regression logistic. If you ...
0
votes
0answers
29 views

PROC ROBUSTREG or PROC NLMIXED in SAS, to down-weight effects of non-normality and outliers

We are conducting an OLS regression in SAS. We have performed all diagnostic tests and concluded that there is non-normality in the residuals, influential outliers and heteroskedasticity. We would now ...
0
votes
2answers
52 views

How do I remove outliers in dataset?

I have a data-set (185 rows) with 20 predictors and 1 dependent variable. I have applied Cook's distance and then 4/N formula to remove some of the outliers in 1st iteration. Should I do this ...
3
votes
1answer
29 views

Do points with high Cook's distance necessarily have a high standardized residual, and vice-versa?

I have two questions below: Could a data point be an influential point if its cook distance is outstanding(greater than 4/(n-p-1)) while its standardised residual is less than 2? It looks like to me ...
0
votes
0answers
16 views

With or without outlier and/or parametric vs non-paramteric

I'm comparing two group on the number or movie seen in the last mont (30 people in each group). Spss say that there 4 outlier but when I'm lookin with the z score I only find 1. If I do a t-test ...
1
vote
0answers
21 views

Winsorizing, just the outlier or all the value?

I have an outlier in my data set. I want to use the winsorizing quartile (to change the outlier to the 5th% and/or 95th%). Looking at the quartiles, sometimes I have more values than just my ...
1
vote
1answer
30 views

Outlier transformation for t-test

In a normally distributed sample with one outlier, some suggest changing the value of the outlier to be one unit above the next highest score. If you are doing a comparison test on two groups using ...
1
vote
1answer
35 views

Robust statistic for representing small dataset with outliers and representing them graphically

I'm evaluating different systems varying a certain parameter common to all of them. Let this parameter be x. At each x value I ...
0
votes
0answers
5 views

Using historic water flux data to detect the existence of leakage, where should I start?

This question is also linked to How to detect a relatively small level shift(leakage) in an hourly water flux time series in an area? which I asked a week ago... Background I've got a series of ...
0
votes
0answers
8 views

Replacing an outlier in one of two terms with a mean

Should I replace values with means? I have an experiment and I know (from the literature) that 3000ms reaction times means the participant is not attentive (the outliers reach 20000ms). I'm analyzing ...
1
vote
1answer
44 views

What comes first: outlier detection or model selection?

I'm fitting a GLMM (mixed logistic regression) in R. I have five covariates. For model selection, I'm using glmmLasso() (in R) to determine which of the five covariates and their interactions should ...
0
votes
1answer
38 views

Finding outliers in multiple dimensions

I'm working on dataset which isn't normally distributed. It contains three dimensions: cost, discount and profit. I'm trying to find outliers in all these dimensions. I used $\text{z-score}$ to find ...
0
votes
1answer
20 views

picking out outliers from a GLM in R

I recently fitted a beta distributed GLM using R (and the betareg package). as you can see the model is a reasonably decent fit, however there are a few outliers. i would like to run the model again ...
0
votes
0answers
28 views

Why leverage measure the distance of the ith observation from the center of the x space? [duplicate]

I know the definition of leverage points in regression, that is $h_{ii}=x_{i}'(X'X)^{-1}x_{i}. $ In many places and text books, they always say that leverage is a standardized measure of the distance ...
3
votes
0answers
19 views

How to detect a relatively small level shift(leakage) in an hourly water flux time series in an area?

Background I'm working on a project which aims to use the history data about a water flux to detect whether there is a leakage happened. The data is hourly collected and among about 4 months. I've ...
0
votes
0answers
29 views

Why different diagnostic tests for detecting outliers in linear regression don't agree with each other?

I have one predictor and one variable, which means that the "cutoff" Mahalanobis value is > 3.84 at 0.05 significance level; covariance ratio should be between 0.88 and 1.12 (1+-3x(1+1)/50); the ...
1
vote
0answers
7 views

Are there kernel-based one-class sparse kernel-based outlier detection methods, e.g. one-class Relevance Vector Machine?

I have a commercial outlier detection problem in moderate dimension (8-25). We have a limited number of true positive tags and can roughly evaluate performance of various methods. So far, the ...
0
votes
0answers
6 views

Intuitive explanation of Grubb's test

Can anyone give an explanation of Grubb's test for outliers? I've found many resources giving steps to calculate it, but none even attempting to talk about why that equation should be a good outlier ...
0
votes
0answers
23 views

How to deal with outliers and feature selection simultaneously?

I've been given some data and need to pick what I consider to be the best features from it and use them to build models that fit the data. My issue is that all the tests I've seen for outliers assume ...
0
votes
1answer
47 views

anomaly/outliers detection in a multilabel dataset on the outcomes

Assuming a multilabel dataset contains a few wrong data. If so, is there a way to predict those wrong outcome given the fact there is a 'pattern' in the predictors? Let's use 'baby and silly' ...
0
votes
0answers
26 views

Outliers detections in time-series

I am searching algorithms for detecting outliers in a time-series data. I see that there are a lot of algorithms and they have an implementation in R. But i don't find any explanation on how they ...
0
votes
0answers
14 views

Which Statistical model should I Choose to fit the Data?

I am struggling with this problem for quite a few days.So far, I have used simple Box-Plot method to pick out the outliers for each location and Diseases. And how can I get outliers after fitting any ...
2
votes
0answers
7 views

Check for outlier in random effects

I have 3 treatements and each treatment is given to 2 random people from which 2 samples is taken from each person that gets the specific treatment. The dataframe in R below shows a sample dataset ...
0
votes
0answers
32 views

Testing for discords in seasonal time series data

I'm trying to find a way to detect discords in seasonal data. I have an algorithm that can select the most likely sub-sequence to be a discord, but what I'm missing is an actual test. I know that ...
1
vote
0answers
32 views

Outlier transformation or no transformation?

I have two group's of 25 people each. I want to know if there a difference between them on a past experience of happiness . I have a test with 100 question in it and all question have an a and b ...
1
vote
0answers
38 views

Linear Regression – When is Bad Data “Too Much”?

I am doing a multiple linear regression analysis. One variable, which I think may be quite predictive, has known bad data. I am currently sampling and using analysts to independently verify the ...
5
votes
1answer
42 views

Cook's distance in detecting outliers

According to my understanding, Cook's distance measures the influence of each observation by excluding points when fitting a model. So I assume it could be an reasonable approach for outlier ...
0
votes
0answers
18 views

Outliers detection or some roubust metrics on long tail sqewed distributions

I have a distribution of user sessions on the web site in the following format date,sessionId,price 2010-01-01,1,0 2010-01-01,2,0 2010-01-01,3,10 ... And I am ...
0
votes
0answers
23 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
28 views

How is finding outliers using extreme value theory different from setting threshold on a pdf of normal events?

Suppose we assume that the data follows a Gaussian distribution. We can set a threshold on its pdf to find outliers. How would it be different from setting a threshold on a pdf, fit the exceedances ...
1
vote
1answer
129 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
1
vote
0answers
11 views

Get rid of values that pollute (or obfuscate) aggregate statistics

I consider myself a beginner in statistical analysis, and I want some advice on a problem that I have encountered in a data analysis task. I am monitoring a system's latency for producing results, ...
2
votes
0answers
54 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
1
vote
0answers
40 views

Use linear regression to detect outliers and leverage points

I want to use linear regression to pre-process the data (e.g find outliers) so that I can use techniques like ANOVA to analyze the data. The goal is not to fit a regression model. I saw two posts ...
1
vote
1answer
106 views

XGBoost (Extreme Gradient Boosting) or Elastic Net More Robust to Outliers

I have recently been doing work with predictive models for a continuous response. I am doing a comparison between Elastic Net (glmnet) package in R and XGBoost ...
3
votes
0answers
30 views

Outlier removal for univariate and multivariate analysis

I have a biological data set on which I would like to do both univariate and multivariate analysis, and try to find correlation of features to a response. Should I remove univariate outliers and do ...
1
vote
0answers
25 views

Stationary time series with outliers

Does anyone have stationary time series data with some outliers from real life? I'd like to try my robust estimation method. Thank you!
2
votes
0answers
25 views

How should I interpret/follow-up on mixed logistic regression (GLMM) diagnostics?

I have experimental data (n subjects = 64) in which the response variable, accuracy (0 or 1), was measured 9 times within subjects. My predictor is Condition (A vs B) measured between subjects. I ...
0
votes
0answers
23 views

Detecting the first outlier in a series of inter-call times

Question: how can I detect the first outlier in a sequence of inter-calls times? Imagine I have a sequence of telephone calls and that the time from call $N$ to call $N-1$ is: lags = {1.1, 1.2, ...
22
votes
6answers
3k views

Is it OK to remove outliers from data?

I looked for a way to remove outliers from a dataset and I found this question. In some of the comments and answers to this question, however, people mentioned that it is bad practice to remove ...
4
votes
3answers
211 views

Sensitivity of the mean to outliers

Is the mean sensitive to the presence of outliers? I initially thought it wasn't, because a small amount of observations shouldn't have much impact, but was told that since those observations have ...
0
votes
0answers
11 views

Method for Outlier Identification in High Dimensions [duplicate]

Is there "state of the art" methods for outlier identification in high dimensions? I have come across PCOut algorithm and a survey paper "A survey on unsupervised outlier detection in ...
0
votes
0answers
10 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
2answers
47 views

Removing outliers from dataset [duplicate]

I am conducting separate regression models for two of my hypotheses. If I spot outliers in one case of conducting the regression model can I then bring them back for my second? Or do these outliers ...
0
votes
0answers
47 views

Hidden Markov Model for anomaly detection

In Hidden Markov Model, it is possible to compute probability of observation sequence by applying forward algorithm given learned model. We can detect anomaly sequences by this algorithm simply by ...
0
votes
0answers
36 views

Regression: using confidence interval to remove outliers

I have to calculate a regression (for instrument calibration purposes) between Log(x) and y, where x is an environmental parameter, known to be log-distributed. Accurate measurement of x is difficult ...