Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

1
vote
2answers
35 views

When forecasting, is it better to remove the outliers or just to transform them?

I am forecasting the number of logins. I have a dataset with the number of logins for each hour. First, I use LOF (local outlier factor) to find the outliers and then I remove them. Second, I use ...
0
votes
1answer
31 views

How to handle timeseries extremes (sigma > 20) in deep learning?

I'm using 16-channel, 400-Hz, standardized EEG data to train CNN-LSTM for seizure classification. The data contains $O(3)$ sigma > 20 points, rarely thousands in a ...
0
votes
0answers
14 views

Likelihood ratio test for mean shift maxima related to background fluctuations [on hold]

I am using the mean-shift algorithm to 'de-blend' clusters resulting from confused (overlapping) sub clusters, in a dataset with noise. Some of the maxima returned by the mean-shift are related to ...
1
vote
0answers
14 views

Is it redundant to normalize test data during Mahalanobis calculation?

I have a training set which is a 12-column dataframe that I'm using to generate a covariance matrix (and possibly a PCA model). My test set is a single 12-column vector. All 12 variables have the ...
1
vote
0answers
27 views

Outlier detection for bivariate bimodal distributions

I have been trying various methods to detect outliers in a bivariate dataset using Mahalanobis in R, but I am unsure about how correct it is since both of my variable vectors are not normally ...
0
votes
0answers
17 views

Removing outlier observation in panel data regression analysis

I have monthly sales data for two year at different stores of a food chain along with some other variables like number of customers, location, customer feedback etc. I am trying to build a panel data ...
1
vote
0answers
24 views

How does the nth sample impact Mean and Variance? [duplicate]

Given the Mean and Variance of $n$ samples $x_i$: $$M_n=\frac {1}{n}\sum_{1}^{n} x_i$$ $$V_n=\frac {1}{n}\sum_{1}^{n}(x_i-μ_n)^2$$ How do Mean and Variance change, when we take into account one ...
0
votes
1answer
21 views

Validation method with outliers

I have a method for prediction in a regression problem with outliers. I'd like to make a validation of my approach. How to make it considering that I need to have outliers in train and test datasets?
1
vote
1answer
66 views

boxplot in R shows wrong outliers

Can someone explain why does boxplot in R show me outliers when they are actually not? I have a dataset for computer sales and I have to predict the price based ...
0
votes
0answers
27 views

Gradient Boosting regression model produces inaccurate forecasts in presence of outliers in a training set [closed]

I'm using XGBoost with a 'reg:linear' objective in order to forecast time series one step forward. I notice that the model appear to start producing forecasts that are too high when there were high-...
2
votes
1answer
72 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
1
vote
1answer
15 views

Robust two-sample test with triplicate measurements?

When testing for a difference in mean between two conditions, biologists typically use a $t$-test, and wring their hands endlessly about how to justify removing outliers. Whereas I typically use a ...
-1
votes
1answer
21 views

Select relevant data to feed model

I am building a regression algorithm to predict the cost of a product defined by several features. I use a supervised rule-based model to do the predictions. The dataset is based on former ...
0
votes
0answers
34 views

Finding outliers in binary data

Say you have data from 10 different sensors about the occurrence of some event - e.g. motion sensors. Each sensor records 1 if they detected the motion sensors and 0 if they didn't. If you have 10 ...
1
vote
0answers
21 views

Outliers and influential observations in elastic net logistic regression

My dataset has many biomarkers and the boxplots of these variables show the presence of many outliers. However, these 'outliers' are real data and not misread observations. I want to use elastic net ...
0
votes
1answer
25 views

Residual vs fitted values (large outlier in predicted value?)

I have attached a plot of residuals vs predicted values for a model i ran. I see 1 large predicted value (extremely large compared to the others) is this a problem for my model? and if so, what can ...
4
votes
0answers
89 views

“Forward search” methods for outlier detection etc. in regression

I am reading Robust Diagnostic Regression Analysis by Anthony Atkinson and Marco Riani. They propose a robust "forward search" method for detecting outliers and other problematic data in regression (...
0
votes
0answers
7 views

How to find out when one of a set of recurrent data is behaving unusually

My problem is finding out a week in which week the data about our sales are "strange". We sell 8 products through a third party vendor V. V sends us, each week, the amount sold for each product. ...
0
votes
0answers
17 views

Detecting intentionally misreported data

I have a dataset containing many intentionally misreported observations. Because I know the base pattern of systematic misreporting I would like to use a systematic procedure to identify and discard ...
1
vote
1answer
35 views

What are the anomalies/fault/outliers detection algorithms

I'm working on a weather application that uses data coming from multiple sensors in real time (the data is time series), i've made an anomalies detection model using One Class Support Vector Machines, ...
3
votes
0answers
26 views

Using robust regression to detect outliers

Rousseeuw and van Zomeren (1990) propose using robust regression to detect multivariate outliers, particularly in OLS regression. This approach seems to make sense (although I have not studied it in ...
1
vote
1answer
32 views

Omitting certain time periods in VAR

I am using a vector autoregression with a monthly lag, and wish to not include certain months, as they are outliers in my analysis and may distort findings. Is estimating such a VAR (using OLS, then ...
0
votes
0answers
18 views

Algorithm to detect outliers in network sensor messages

I have a network sensor device which generates a number of messages. The message is of format "timeofmessage messagetype messageimportance messagetext". The sensor keeps producing "sensor-ok" messages ...
3
votes
2answers
46 views

How to identify outliers in a time series with correlated variables

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the ...
0
votes
2answers
34 views

Dealing with extreme outliers in administrative data (in R)

I work with some data that includes some "extreme outliers". E.g. timestamps that are totally unreasonable (surgery took 20 days when most take 1 hour). Is there a set of principles one can use to ...
0
votes
0answers
20 views

Outlier Detection using Dixon’s Q-Test

I am trying to identify outliers in a small dataset. I read that Dixton's Q-test is useful for such case. Here is a refreshment of Dixton's Q-test: https://chem.libretexts.org/Ancillary_Materials/...
0
votes
0answers
28 views

SVM one class outlier detection/classifier with Matlab

I have several independent time series (a small sample is in the end of the question) and I am trying to find the outliers using SVM. I have used this to find the outliers ...
0
votes
0answers
14 views

Is a single count data point significantly different from a set of reference count data

I have counted the number of occurrences of protein domains in the proteomes of 16 different insect species, 1 focal vs 15 reference species. For each of the ~4700 protein domains in my data, I have a ...
2
votes
1answer
102 views

Why is lasso more robust to outliers compared to ridge?

In my attempt to reason about it intuitively I am concluding that ridge might be more robust to outliers. Following is my intuitive/lose reasoning : If there is an outlier then to match my ...
0
votes
0answers
23 views

finding regression coefficients and deviation with autocorrellated outlaws

I try to make regression analyses to vector of average month C02 concentration in the air. ...
0
votes
1answer
29 views

When does it make sense to detect multivariate outliers instead of univariate ones?

I do get the idea of univariate outliers and detecting them. However, I don't understand the idea of multivariate outliers. More precisely, I would like to ask if detecting multivariate outliers only ...
0
votes
0answers
59 views

Detecting trend in panel data, smoothing techniques and outlier detection

I'm conducting an analysis on a Landsat scene to detect trends for change detection phenomena (forest disturbances) over a time series of 20 years. I identified on the image the pixels that are ...
1
vote
1answer
16 views

Spikes impact on time series stationarity

I have a demand time series that are highly impacted by promotion (spikes). Do spikes violate the assumptions of stationarity? Can we apply the KPSS test or ADF to test whether the series is ...
0
votes
0answers
15 views

Outliers in Regression & Scaling

How do I handle outliers in linear regression? I have been flagging outliers as a 1 and every other row in my data set as a 0. But this means that my column is binary and scales very differently ...
0
votes
0answers
28 views

Workflow in data preparation with Box-Cox transformation

I have a dataset with both missing values and outliers in continuous features. I would like to perform Box-Cox transformation on every continuous feature to reach the best distribution. Box-Cox works ...
1
vote
1answer
27 views

Query regarding analysis of regression output

I have made 2 models from based on some social media data.First one is with outliers and the 2nd one is without outliers. ...
0
votes
0answers
47 views

Winsorizing data in small sample

I have a relatively small sample of panel data (quarterly data for 68 firms over 7 years). My dependent variable is positively skewed. In order to limit the influence of observations with large values,...
0
votes
0answers
20 views

determine suitable values for the parameters of the distance function for this graph

Hi I've been learning data mining and came across this question. I couldn't seem to figure it out myself. So we have a large single undirected graph(without attributes) G = (V,E) and want to detect ...
0
votes
0answers
30 views

Inserting into a data analysis the probability of being an outlier

This question is a bit generic. Let's suppose we have some quantities measured $X_i$ and we want to estimate some quantity, for example $\mu$, the mean of the distribution. For every quantity we ...
0
votes
3answers
371 views

Check statistical significance of one observation [duplicate]

I have got a dataset and I need to check if one specific observation within the dataset is significantly different from the dataset's mean. ...
0
votes
1answer
21 views

How to find categorical contributing factors for an anomaly?

Given a house sales dataset with number of houses sold each day and their attributes (i.e., price, number of rooms, size, etc.) - if on a specific day there's a spike/drop in sales, what are some ...
0
votes
0answers
8 views

Outliers odd behaviour

I got a sample of data. I applied a linear correction to it and then I calculate again the mean and standard deviation. The mean and the standard deviation before and after the correction are almost ...
2
votes
1answer
39 views

A regressor failed to learn extreme values

I am working on a regression problem using xgbclassifier (https://xgboost.readthedocs.io/en/latest/python/python_api.html) The output values range from 0 to 10 (log-normal distribution), but when I ...
0
votes
0answers
185 views

Removing outliers in logistic regression

I am running a logistic regression analysis to model if a patient has a specific disease or not. I want to remove outliers because i want my model to be as accurate as possible. For the same I learnt ...
0
votes
1answer
46 views

Leverage and Influence

Is it possible that an outlier is neither influential nor does it have high leverage? Or can it happen that an observation with high leverage is not an outlier and is neither influential?
1
vote
0answers
115 views

modified z score

I am using Modified Z-Score to find out outliers on a time series data on exit rate for a website. N = 1131. Based on last 3 years daily data (1096 values), i am finding out outliers for the ...
0
votes
0answers
21 views

How to perform feature scaling on noise removel process?

i'm working on dataset contain machinery sensor data. each column(feature) represent different sensor data(pressure, temperature, speed, etc) of the machine part. here task is to predict normal ...
2
votes
1answer
72 views

Comparing two time series statistically? [closed]

This question has been asked before with very good (but incomplete) answers. This and this are the two best answers that I found. But following is my doubt: Top answers from both (by IrishStat) the ...
1
vote
1answer
177 views

Why do we use squared deviations to compute the SD, given that it amplifies the effect of outliers? [duplicate]

Suppose I have the following hypothetical data: One thousand times value 15 (i.e., 15 occurs 1000 times) and a single outlier value - 115 (i.e., 115 occurs just once - an outlier) Thus the mean is: $...
0
votes
0answers
7 views

unbalanced due to regimes?

I have a sample which seems to have data in two distinct regimes. Suppose that 50% of observations have x variable from 0 to 1, while the remaining 50% with x between 1 and 8. Y appears to increase ...