Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

0
votes
0answers
20 views

finding regression coefficients and deviation with autocorrellated outlaws

I try to make regression analyses to vector of average month C02 concentration in the air. ...
0
votes
1answer
20 views

When does it make sense to detect multivariate outliers instead of univariate ones?

I do get the idea of univariate outliers and detecting them. However, I don't understand the idea of multivariate outliers. More precisely, I would like to ask if detecting multivariate outliers only ...
0
votes
0answers
21 views

Detecting trend in panel data, smoothing techniques and outlier detection

I'm conducting an analysis on a Landsat scene to detect trends for change detection phenomena (forest disturbances) over a time series of 20 years. I identified on the image the pixels that are ...
0
votes
1answer
7 views

Spikes impact on time series stationarity

I have a demand time series that are highly impacted by promotion (spikes). Do spikes violate the assumptions of stationarity? Can we apply the KPSS test or ADF to test whether the series is ...
0
votes
0answers
11 views

Outliers in Regression & Scaling

How do I handle outliers in linear regression? I have been flagging outliers as a 1 and every other row in my data set as a 0. But this means that my column is binary and scales very differently ...
0
votes
0answers
24 views

Workflow in data preparation with Box-Cox transformation

I have a dataset with both missing values and outliers in continuous features. I would like to perform Box-Cox transformation on every continuous feature to reach the best distribution. Box-Cox works ...
1
vote
1answer
27 views

Query regarding analysis of regression output

I have made 2 models from based on some social media data.First one is with outliers and the 2nd one is without outliers. ...
0
votes
0answers
6 views

What statistical test should I use when testing many factorial ID and one continuous D, with n = 22,875 [closed]

I need some advice on what statistical tests I should be doing with this large quantity of data. 4 independent factorial variables: location, habitat, direction, species ID. I also have time, if ...
0
votes
0answers
23 views

Winsorizing data in small sample

I have a relatively small sample of panel data (quarterly data for 68 firms over 7 years). My dependent variable is positively skewed. In order to limit the influence of observations with large values,...
0
votes
0answers
14 views

determine suitable values for the parameters of the distance function for this graph

Hi I've been learning data mining and came across this question. I couldn't seem to figure it out myself. So we have a large single undirected graph(without attributes) G = (V,E) and want to detect ...
0
votes
0answers
30 views

Inserting into a data analysis the probability of being an outlier

This question is a bit generic. Let's suppose we have some quantities measured $X_i$ and we want to estimate some quantity, for example $\mu$, the mean of the distribution. For every quantity we ...
0
votes
3answers
151 views

Check statistical significance of one observation [duplicate]

I have got a dataset and I need to check if one specific observation within the dataset is significantly different from the dataset's mean. ...
0
votes
1answer
19 views

How to find categorical contributing factors for an anomaly?

Given a house sales dataset with number of houses sold each day and their attributes (i.e., price, number of rooms, size, etc.) - if on a specific day there's a spike/drop in sales, what are some ...
0
votes
0answers
8 views

Outliers odd behaviour

I got a sample of data. I applied a linear correction to it and then I calculate again the mean and standard deviation. The mean and the standard deviation before and after the correction are almost ...
2
votes
1answer
34 views

A regressor failed to learn extreme values

I am working on a regression problem using xgbclassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html) The output values range from 0 to 10 (log-normal distribution), but when I ...
0
votes
0answers
72 views

Removing outliers in logistic regression

I am running a logistic regression analysis to model if a patient has a specific disease or not. I want to remove outliers because i want my model to be as accurate as possible. For the same I learnt ...
1
vote
0answers
48 views

modified z score

I am using Modified Z-Score to find out outliers on a time series data on exit rate for a website. N = 1131. Based on last 3 years daily data (1096 values), i am finding out outliers for the ...
0
votes
0answers
16 views

How to perform feature scaling on noise removel process?

i'm working on dataset contain machinery sensor data. each column(feature) represent different sensor data(pressure, temperature, speed, etc) of the machine part. here task is to predict normal ...
1
vote
0answers
32 views

Comparing two time series statistically?

This question has been asked before with very good (but incomplete) answers. This and this are the two best answers that I found. But following is my doubt: Top answers from both (by IrishStat) the ...
1
vote
1answer
174 views

Why do we use squared deviations to compute the SD, given that it amplifies the effect of outliers? [duplicate]

Suppose I have the following hypothetical data: One thousand times value 15 (i.e., 15 occurs 1000 times) and a single outlier value - 115 (i.e., 115 occurs just once - an outlier) Thus the mean is: $...
0
votes
0answers
6 views

unbalanced due to regimes?

I have a sample which seems to have data in two distinct regimes. Suppose that 50% of observations have x variable from 0 to 1, while the remaining 50% with x between 1 and 8. Y appears to increase ...
0
votes
1answer
33 views

Using outlier records as a feature in model building

I am exploring the Big Mart Sales III dataset and trying to understand if using outlier rows to build a feature for predictive modeling is a sound and correct approach. This is how I have proceeded ...
3
votes
1answer
66 views

Outliers on discrete data

Is there any robust methodology to identify outliers in the discrete data distribution. I am specifically concerned with discrete geometrical distribution? P.S. Data transformation does not seem to ...
9
votes
2answers
449 views

What are “fringeliers”?

I recently received a reviewer comment from a journal submission that asked me to report how I dealt with outliers and fringeliers. I had not heard of the term "fringeliers" and when I googled, ...
0
votes
0answers
21 views

Which statistical tests are used for determining if an LSRL point is an outlier or influential?

Question basically in the title. For example, given the entire dataset of an LSRL as points I'm trying to find which points are considered outliers and which are considered influential points using ...
1
vote
0answers
19 views

IQR based outlier detection with multivariate data

One method to detect outliers in a dataset $[x_1 ... x_N], x_i \in R$ consists in finding the samples $x_i$ such that $$ x_i \lt Q_1-K*IQR | x_i \gt Q3 + K*IQR $$ where $Q_1$ and $Q_3$ are the first ...
1
vote
0answers
8 views

Replacing very high/low observation differences w/ averages to create an adjusted time series a good outlier adjustment method for time series data?

I have a monthly time series that stretches about 18 years. I examine the over-the-month differences in a time series and observe that 3 or 4 OTM values are very extreme. I can identify these values ...
0
votes
1answer
73 views

Noise and Outliers in DBSCAN

Why are noise and outliers treated as the same concept in DBSCAN (density-based spatial clustering of applications with noise)?
1
vote
0answers
17 views

Open implementation of Xu, Caramis and Mannor's outlier-robust PCA?

The answer linked below discusses an outlier-tolerant PCA method. Is there a publicly available implementation? https://stats.stackexchange.com/a/71928/86176 Here's the paper: Xu, H., Caramanis, C....
3
votes
2answers
49 views

Can linear SVM classify samples if there is no difference in means of predictors?

Let's say we have a standard classification problem where we want to classify samples into two groups based on some number of predictors. Is it possible to do this with above-chance accuracy, if ...
1
vote
0answers
31 views

Test to determine whether the empirical distribution for a given day is an outlier compared with other days

Say you have multiple data samples from different days (or some other unit of time) and you want to answer the question: is the distribution for a given day an outlier (compared to other days)? Is ...
0
votes
0answers
43 views

How outliers influence your results? and what are good and bad leverage points? [duplicate]

I am confused between outliers and leverage points. And the difference between good and bad leverage points in time series analysis. Can somebody help me?
0
votes
1answer
119 views

finding outliers in mixed model [duplicate]

I'm trying to find outliers in this mixed model: m1 <- lmer(y ~ service + lectage + studage + (1|d) + (1|s), data=InstEval) So I used the ...
0
votes
1answer
371 views

Feature Importance in Isolation Forest

In an unsupervised setting for higher-dimensional data (e.g. 10 variables (numerical and categorical), 5000 samples, ratio of anomalies likely 1% or below but unknown) I am able to fit the isolation ...
0
votes
0answers
30 views

LocalOutlierFactor scikit-learn

My goal is to use the LocalOutlierFactor class from scikit-learn to do real-time Novelty Detection. This can be achieved by setting novelty=True in the constructor, ...
0
votes
0answers
31 views

Accurate Prediction of Rare values in Regression

I am working on a project which is to determine Systolic and Diastolic Blood Pressure from a set of Independent variables. One of the issues that I am facing is predicting rare values (Hypotensive and ...
0
votes
0answers
47 views

biological basis for excluding values outside 3 standard deviations from the mean? [duplicate]

Is there any biological basis for excluding outliers in a dataset of blood cytokine levels (e.g. values outside 3 standard deviations from the mean for each cytokine) as measured by multiplex ...
0
votes
0answers
22 views

Modeling relationships between 100+ variables

I have been interested in DS/ML for a few years now and I have been able to build some relatively simple models actually performing pretty well. Now I have this idea in mind but I am not sure how to ...
1
vote
1answer
412 views

Can One-Class SVM be used for outlier detection?

According to my readings (Support Vector Method for Novelty Detection, for instance), One-Class SVM can be used for novelty detection only. The purpose of the $\nu$ parameter is to defined the maximum ...
0
votes
0answers
100 views

Adaptive threshold setting for parametric anomaly detection system applied to time series data

I just started my first project where I'm trying to find anomalies in the energy usage of a air conditioner. The only usable data I could obtain was the energy data for a few months. Since the energy ...
0
votes
0answers
13 views

Is it appropriate to perform outlier treatment on test sample data set? I am building logistic regression model

I am building logistic regression model. Is it ok to perform outlier treatment on significant variables after building the model and if yes, do we need to perform outlier treatment on test sample data ...
1
vote
1answer
40 views

How to detect outlier samples in gene expression studies?

I have a matrix of n observations where each observation has m variables. So I am faced in with a matrix of mxn. How may I determine which observations are outliers? Thank you in advance.
0
votes
0answers
20 views

Determine “correct” data based on multiple sources

I tried searching for this but might be missing some important keywords as I could not find anything. Data Available : I have 1 to 4 sources of data showing temperature for various locations on a ...
0
votes
0answers
55 views

What are some fast outlier detection methods for big data in R?

I have a large dataset (300,000 rows) for which there are clear outliers. Box plots of two of the DVs of interest reveal the presence of large numbers of outliers by the Tukey outlier detection rule (...
4
votes
0answers
78 views

Can I statistically describe a single case/outlier vs. a distribution?

I have a dataset consisting of body weight and corresponding age for a bunch of healthy subjects (grey triangles below). I fit a nonlinear function to this data and graphed a 95% prediction interval. ...
1
vote
0answers
53 views

R code for robust ridge regression

I am having trouble in searching for the MSE value in using robust ridge regression. The robust estimators that i used is LTS and MM. However, when both robust estimators were applied to ridge, the ...
0
votes
1answer
29 views

Get rid of the irrelevant points [closed]

Take some time to look at the picture above. We can notice a red cloud. There's a red dense cloud and some irrelevant red points around that red cloud. Suppose the red cloud to be the set of vector $...
1
vote
1answer
64 views

Numerically Distinguish Between Real Correlation and Artifact

I'm looking at correlation for a large number of vectors, and many (about 3000) of these pairwise comparisons appear to have a significant correlation even after Bonferroni correction. Plotting these ...
0
votes
0answers
22 views

How does Outliers affect logistic regression? [duplicate]

I see this is answered here : How does outlier impact logistic regression? But I am not able to understand how does it affect logistic regression , can anyone take a step back and explain in a bit ...
1
vote
1answer
55 views

How to calculate the standard average of a set excluding outliers? [closed]

I have a set of numbers, and I need to calculate their average excluding outlier values (which I don't know a priori). It came to mind that many years ago I studied Standard Deviation. Could I apply ...