Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

3
votes
1answer
55 views

Outliers on discrete data

Is there any robust methodology to identify outliers in the discrete data distribution. I am specifically concerned with discrete geometrical distribution? P.S. Data transformation does not seem to ...
0
votes
0answers
13 views

How to construct envelope graphics for residuals? [on hold]

Using R we can plot the envelope graphic for regression residuals using the function qqPlot. My question is: How to construct ...
0
votes
1answer
34 views

How to handle outlier in machine learning? [closed]

What is the best approach to handle the outliers in the data. when we have to remove the outliers and when i have to change the outliers values to make them within the range. it will be helpful if ...
8
votes
2answers
431 views

What are “fringeliers”?

I recently received a reviewer comment from a journal submission that asked me to report how I dealt with outliers and fringeliers. I had not heard of the term "fringeliers" and when I googled, ...
0
votes
0answers
18 views

Which statistical tests are used for determining if an LSRL point is an outlier or influential?

Question basically in the title. For example, given the entire dataset of an LSRL as points I'm trying to find which points are considered outliers and which are considered influential points using ...
1
vote
0answers
7 views

IQR based outlier detection with multivariate data

One method to detect outliers in a dataset $[x_1 ... x_N], x_i \in R$ consists in finding the samples $x_i$ such that $$ x_i \lt Q_1-K*IQR | x_i \gt Q3 + K*IQR $$ where $Q_1$ and $Q_3$ are the first ...
1
vote
0answers
8 views

Replacing very high/low observation differences w/ averages to create an adjusted time series a good outlier adjustment method for time series data?

I have a monthly time series that stretches about 18 years. I examine the over-the-month differences in a time series and observe that 3 or 4 OTM values are very extreme. I can identify these values ...
0
votes
1answer
19 views

Noise and Outliers in DBSCAN

Why are noise and outliers treated as the same concept in DBSCAN (density-based spatial clustering of applications with noise)?
0
votes
0answers
8 views

Open implementation of Xu, Caramis and Mannor's outlier-robust PCA?

The answer linked below discusses an outlier-tolerant PCA method. Is there a publicly available implementation? https://stats.stackexchange.com/a/71928/86176 Here's the paper: Xu, H., Caramanis, C....
0
votes
0answers
16 views

How detect outliers with get.knn

I am trying to detect outliers in a multivariate data with R using get.knn() function from the ...
3
votes
2answers
47 views

Can linear SVM classify samples if there is no difference in means of predictors?

Let's say we have a standard classification problem where we want to classify samples into two groups based on some number of predictors. Is it possible to do this with above-chance accuracy, if ...
1
vote
0answers
28 views

Test to determine whether the empirical distribution for a given day is an outlier compared with other days

Say you have multiple data samples from different days (or some other unit of time) and you want to answer the question: is the distribution for a given day an outlier (compared to other days)? Is ...
0
votes
0answers
21 views

How outliers influence your results? and what are good and bad leverage points?

I am confused between outliers and leverage points. And the difference between good and bad leverage points in time series analysis. Can somebody help me?
-1
votes
0answers
19 views

Data smoothing through binning?

I have a small dataset of size n=26. ...
0
votes
1answer
39 views

finding outliers in mixed model [duplicate]

I'm trying to find outliers in this mixed model: m1 <- lmer(y ~ service + lectage + studage + (1|d) + (1|s), data=InstEval) So I used the ...
0
votes
0answers
114 views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
0
votes
0answers
14 views

LocalOutlierFactor scikit-learn

My goal is to use the LocalOutlierFactor class from scikit-learn to do real-time Novelty Detection. This can be achieved by setting novelty=True in the constructor, ...
0
votes
0answers
28 views

Accurate Prediction of Rare values in Regression

I am working on a project which is to determine Systolic and Diastolic Blood Pressure from a set of Independent variables. One of the issues that I am facing is predicting rare values (Hypotensive and ...
0
votes
0answers
44 views

biological basis for excluding values outside 3 standard deviations from the mean? [duplicate]

Is there any biological basis for excluding outliers in a dataset of blood cytokine levels (e.g. values outside 3 standard deviations from the mean for each cytokine) as measured by multiplex ...
0
votes
0answers
20 views

Modeling relationships between 100+ variables

I have been interested in DS/ML for a few years now and I have been able to build some relatively simple models actually performing pretty well. Now I have this idea in mind but I am not sure how to ...
1
vote
1answer
169 views

Can One-Class SVM be used for outlier detection?

According to my readings (Support Vector Method for Novelty Detection, for instance), One-Class SVM can be used for novelty detection only. The purpose of the $\nu$ parameter is to defined the maximum ...
0
votes
0answers
50 views

Adaptive threshold setting for parametric anomaly detection system applied to time series data

I just started my first project where I'm trying to find anomalies in the energy usage of a air conditioner. The only usable data I could obtain was the energy data for a few months. Since the energy ...
0
votes
0answers
12 views

Is it appropriate to perform outlier treatment on test sample data set? I am building logistic regression model

I am building logistic regression model. Is it ok to perform outlier treatment on significant variables after building the model and if yes, do we need to perform outlier treatment on test sample data ...
1
vote
1answer
31 views

How to detect outlier samples in gene expression studies?

I have a matrix of n observations where each observation has m variables. So I am faced in with a matrix of mxn. How may I determine which observations are outliers? Thank you in advance.
0
votes
0answers
16 views

Determine “correct” data based on multiple sources

I tried searching for this but might be missing some important keywords as I could not find anything. Data Available : I have 1 to 4 sources of data showing temperature for various locations on a ...
0
votes
0answers
32 views

What are some fast outlier detection methods for big data in R?

I have a large dataset (300,000 rows) for which there are clear outliers. Box plots of two of the DVs of interest reveal the presence of large numbers of outliers by the Tukey outlier detection rule (...
4
votes
0answers
74 views

Can I statistically describe a single case/outlier vs. a distribution?

I have a dataset consisting of body weight and corresponding age for a bunch of healthy subjects (grey triangles below). I fit a nonlinear function to this data and graphed a 95% prediction interval. ...
1
vote
0answers
38 views

R code for robust ridge regression

I am having trouble in searching for the MSE value in using robust ridge regression. The robust estimators that i used is LTS and MM. However, when both robust estimators were applied to ridge, the ...
0
votes
1answer
27 views

Get rid of the irrelevant points [closed]

Take some time to look at the picture above. We can notice a red cloud. There's a red dense cloud and some irrelevant red points around that red cloud. Suppose the red cloud to be the set of vector $...
1
vote
1answer
40 views

Numerically Distinguish Between Real Correlation and Artifact

I'm looking at correlation for a large number of vectors, and many (about 3000) of these pairwise comparisons appear to have a significant correlation even after Bonferroni correction. Plotting these ...
0
votes
0answers
21 views

How does Outliers affect logistic regression? [duplicate]

I see this is answered here : How does outlier impact logistic regression? But I am not able to understand how does it affect logistic regression , can anyone take a step back and explain in a bit ...
1
vote
1answer
50 views

How to calculate the standard average of a set excluding outliers? [closed]

I have a set of numbers, and I need to calculate their average excluding outlier values (which I don't know a priori). It came to mind that many years ago I studied Standard Deviation. Could I apply ...
21
votes
1answer
3k views

Why is PCA sensitive to outliers?

There are many posts on this SE that discuss robust approaches to principal component analysis (PCA), but I cannot find a single good explanation of why PCA is sensitive to outliers in the first place....
0
votes
0answers
20 views

Removing an outlier in a single measure in multivariate data

We have recorded some kinematic data, and are looking at three measures derived from the movement data for each subject. In order to identify outliers, we are using the Mahanalobis distance. I got ...
0
votes
0answers
7 views

Is there a function to calculate the critical Tietjen-Moore value?

The NIST article on the Tietjen-Moore Test for outliers recommends calculating the critical value by simulation, generating 10,000 sets of data. Is there a function that can be numerically integrated ...
0
votes
1answer
66 views

Should you normalize your training data for Local Outlier Factor

Say I'm using scikits implementation of Local Outlier Factor with euclidean distance being used by the reachability function. My input features are magnitudes apart, so is it advisable that I ...
2
votes
0answers
59 views

For a small dataset, should I apply an adjustment to the 3-sigma rule when using it for outlier detection?

Standard deviations for small datasets (for example, less than 20 data points) are more volatile/higher variance than those for large datasets. If we want to use the 3-sigma rule for outlier detection,...
0
votes
0answers
121 views

Using ARIMA with exogenous regressors for outlier detection in R

I would like to detect outliers in real-time data that is aggregated per hour. For this example, I've selected the hourly pedestrian data from Melbourne, Australia (Pedestrian volume (updated monthly),...
0
votes
1answer
34 views

Is there any paper shown how to solve an one class SVM by SMO type algorithm

The one class SVM can be used as an outlier rejection. An example can be found on. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3041295/ Generally one class SVM is shown as a constrained quadratic ...
1
vote
0answers
27 views

What regression diagnostics should I perform for an ordered probit?

Currently I have done the following diagnostics with the linktest multicollinearity with vif the parallel lines assumption with lr test of the oprobit and goprobit. I have seen that I may have to ...
0
votes
1answer
44 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
1
vote
0answers
19 views

Name for spurious linear Regression Plots

Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
0
votes
0answers
18 views

Finding Thresholds for Sales Outliers

I am trying to determine the best methods for analyzing future data in terms of the prior 5 years worth of sales data. The data is sales related so when I plot out the transactions it is highly skewed ...
0
votes
1answer
28 views

Best practice for outlier removal in Investigating a process deviation

In a controlled process, in which a specific product depicted a deviation in a final product result. The process is time controlled, in which the historical manufacturing experience of the product is ...
0
votes
0answers
22 views

Correct Usage of IsolationForest

I am working on a Regression problem. I want to remove outliers before building a model, and I narrowed down to IsolationForest (scikit-learn implementation) as it is high dimensional data. I got ...
0
votes
1answer
22 views

Clustering data with outliers without ignoring them

I have 2 dimensional data about customer churn,i Clustered them with k-means but because of Outliers the clusters are not homogeneous.actually Outliers are about customers which are very important , ...
2
votes
2answers
65 views

Outlier and correlation

Hi, I have a question. The scatter plot doesn't show any type of correlation and there is an outlier. If the outlier was to be removed, would the correlation: Increase dramatically Increase ...
0
votes
0answers
18 views

Tukey’s rule for outlier analysis: What does it mean when the HIGH is below the mean?

Hi as you know the upper boundary of Tukey's rule is Upper Range = Q3 + (1.5 * IQR) however recently I came across something that does not make sense to me. ...
0
votes
0answers
76 views

What are the different influences of outliers regarding the feature scaling methods: standardization VS. normalization?

I've come to know that normalization (MinMax scaling) and standardization (Z-score normalization) on data have different influences from outliers in the data. In About Feature Scaling and ...
2
votes
2answers
194 views

Does the presence of the outliers affect the 1NN algorithm?

I am working on KNN algorithm. I uploaded and prepared the following dataset. ...