Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

Filter by
Sorted by
Tagged with
-1
votes
1answer
56 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
0
votes
1answer
35 views

What is the method should I use to calculate similarity in a data set with outliers that must be included?

Information: So I have a data set with 18 vectors with 167 components, each of with has a value with a range of $[-2, 2]$. I am trying to calculate the similarity between one arbitrary vector in ...
2
votes
1answer
66 views

Detecting extrema under uncertainty?

An old version of this question was poorly articulated. Here is another go: I have fifty objects. With a different, independent, unbiased scale for each object, I measure their weights 100 times each ...
2
votes
1answer
23 views

Minimum generalized variance for outlier detection

I'm currently reading the paper Distribution of Variables by Method of Outlier Detection and am trying to understand the section on Minimum generalized variance If ...
0
votes
0answers
18 views

Simulate multivariate outliers

Considering a multivariate linear model $\boldsymbol{Y = XB + E}$, where $\boldsymbol{Y, X, B}$ and $\boldsymbol{E}$ have dimension $n \times m$, $n \times p$, $p \times m$ and $n \times m$, ...
-1
votes
2answers
37 views

In elbow curve how to find the point from where the curve starts to rise? [closed]

I am computing a distance on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be discarded. ...
0
votes
1answer
31 views

have new outliers after capping

I'm trying to cap outliers in a column my pandas DataFrame. Here's the boxplot for a column of my original data. boxplot for a column of my original data So, using code from this stackoverflow answer, ...
2
votes
3answers
61 views

What would be the outliers in this specific situation? (given in description)

Recently over a phone interview I was asked this: "I have data consisting of 102 points between 0 and 100 where the first 50 points, starting at 0, are incremented by 0.5 every time. The next two ...
1
vote
1answer
27 views

Outliers of small dataset

I have a python function that takes a list of smaller images boxes (represented as float arrays) and the whole image img in as a ...
-1
votes
0answers
27 views

Outlier treatment with Interquartile range and filling missing values [python]

Previously, I did manual removal of outliers and did not use any outlier detection methods. ...
-1
votes
0answers
24 views

Detecting outlier detection using unsupervised learning

Problem is to find server outlier detection from cluster of 100 servers. My data looks like: I have every 5 minutes of data snapshot column data: time ---- cpu metrics ----- host_name I am ...
0
votes
0answers
61 views

Assessing a binary decission based on continuos and multi-level categorical variables

I have been asked to generate a tool to assess if a particular new set of measurements fit within a list of already accepted ones. The problem is that there are different categorical variables with ...
2
votes
0answers
16 views

Can cross validation be applied to threshold based outlier detection model?

I have a threshold based outlier detection model. I apply PCA then calculate the distance from the centre of the features, and use the MSE to differentiate if the datapoint is a outlier. However, I ...
0
votes
0answers
17 views

What is the relationship between negative autocorrelation and first differenced outliers?

I am currently working with economic data and I am trying to model my dependent variable using several macroeconomic independent variables. When I tried linear regression on the data, I got a high R-...
0
votes
0answers
9 views

Estimation of AO effect based on least squares theory

Given the following model, capturing the presence of an additive outlier in a time series, where $T$ is the time when the additive outlier is recorded and $w_A$ is its effect, $a_t$ is a white-noise ...
0
votes
1answer
30 views

Normality Tests in samples with outliers

I'm making a code in R that contains some parametric and non-parametric tests, like ANOVA and Kruskal-Wallis. To know if I should use one or another I check the "normality" of the test sample. My ...
1
vote
1answer
22 views

scikit-learn IsolationForest no variance feature

I'm using IsolationForest algorithm in order to detect anomalies in my data and to use this model to detect future anomalies in new rows and came across a few questions: Is the model good for ...
0
votes
1answer
28 views

Clarification in Anomaly Detection Algorithm

I am referring to Prof Andrew Ng Coursera ML notes (Week 9). He says that to identify outliers we first model the training data and then fit a Gaussian distribution with probability density $ p(X; \mu ...
0
votes
0answers
11 views

Setting Choosing alpha for generalized extreme Studentized deviate ESD

I'm working on a S-H-ESD implementation and I'm struggling to set the alpha for the ESD. The suggested alpha everywhere is 0.05. Is there a way to calculate an alpha based on the expected percent of ...
1
vote
0answers
24 views

Removing Outliers From Non-Linear Data in the Inappropriate Way Gives a Better Result

I'm doing a cost prediction model for mechanic parts (12000 rows and 43 variables). All the numeric variable(10 numeric variables total) shows no normal distribution. After the log transformation, ...
13
votes
1answer
882 views

Dropping outliers based on “2.5 times the RMSE”

In Kahneman and Deaton (2010)$^\dagger$, the authors write the following: This regression explains 37% of the variance, with a root mean square error (RMSE) of 0.67852. To eliminate outliers and ...
1
vote
1answer
26 views

What outlier score is used here?

I have come across a score function in a program, but I don't exactly understand what it does. This score is a measure of how probable a sampled/created value is. I will describe the procedure for ...
0
votes
1answer
42 views

Coffee ratings: How to do causal inference with high kurtosis/outliers?

I have data (n>1000) on ratings of various coffee features, and then a final overall score. I am interested in inference: What is the effect of a feature on overall score. However, my data has ...
1
vote
1answer
32 views

Is it OK to have only a single class labels in test data for prediction with one-class-svm?

I have a data which has only a single class, namely, '0'. There is no 'not 0' class. The one-class SVM model was trained on a <...
0
votes
1answer
48 views

Should you standardize your variables before or after removing outliers?

Barring the question of how to operationalize outliers, or the utility of doing so, and assuming dependent variables and independent variables are all scaled in the main regression specification (...
0
votes
0answers
13 views

Log-Normalization of skewed data before feeding to neural network models ( autoencoders)

If your input data has few columns that are extremely skewed, It is well known that one would log normalize ( take log and then normalize or standardize) the data before passing to regression ...
0
votes
0answers
61 views

Unsupervised anomaly and outlier detection of database queries

I'm monitoring database queries coming from multiple different applications spread across numerous systems and I'd like to find both anomalous queries as well as outliers in a completely unsupervised ...
3
votes
1answer
344 views

Is it good to plot a scatterplot with weird regression line? [closed]

Am a beginner in data analysis and i feel that something about these graphs are wrong. Am not sure that the outliers are the issue or am doing this the wrong way. Thanks for the help
2
votes
0answers
23 views

Detecting outliers in time series with fitting curve

I have a collection of time series that I know follow a particular type of non-linear functions. For each time series I use the LM algorithm to obtain a fitting curve. I have already checked that the ...
2
votes
1answer
62 views

Detecting outliers in time-series if I don't have a “normal” dataset [duplicate]

I have been trying to detect anomalies in my time-series dataset. What I am trying to accomplish is the following: I have a multivariate dataset, where two columns are of special interest. One tells ...
1
vote
2answers
59 views

When forecasting, is it better to remove the outliers or just to transform them?

I am forecasting the number of logins. I have a dataset with the number of logins for each hour. First, I use LOF (local outlier factor) to find the outliers and then I remove them. Second, I use ...
0
votes
1answer
54 views

How to handle timeseries extremes (sigma > 20) in deep learning?

I'm using 16-channel, 400-Hz, standardized EEG data to train CNN-LSTM for seizure classification. The data contains $O(3)$ sigma > 20 points, rarely thousands in a ...
1
vote
0answers
19 views

Is it redundant to normalize test data during Mahalanobis calculation?

I have a training set which is a 12-column dataframe that I'm using to generate a covariance matrix (and possibly a PCA model). My test set is a single 12-column vector. All 12 variables have the ...
1
vote
0answers
40 views

Outlier detection for bivariate bimodal distributions

I have been trying various methods to detect outliers in a bivariate dataset using Mahalanobis in R, but I am unsure about how correct it is since both of my variable vectors are not normally ...
0
votes
0answers
19 views

Removing outlier observation in panel data regression analysis

I have monthly sales data for two year at different stores of a food chain along with some other variables like number of customers, location, customer feedback etc. I am trying to build a panel data ...
1
vote
0answers
24 views

How does the nth sample impact Mean and Variance? [duplicate]

Given the Mean and Variance of $n$ samples $x_i$: $$M_n=\frac {1}{n}\sum_{1}^{n} x_i$$ $$V_n=\frac {1}{n}\sum_{1}^{n}(x_i-μ_n)^2$$ How do Mean and Variance change, when we take into account one ...
0
votes
1answer
26 views

Validation method with outliers

I have a method for prediction in a regression problem with outliers. I'd like to make a validation of my approach. How to make it considering that I need to have outliers in train and test datasets?
1
vote
1answer
81 views

boxplot in R shows wrong outliers

Can someone explain why does boxplot in R show me outliers when they are actually not? I have a dataset for computer sales and I have to predict the price based ...
2
votes
1answer
73 views

Finding Outliers in Resource Allocation Forecast Data

I initially posted this under the DS stack exchange, but after much reading and browsing, I think this is the right place for this question. I'm a workforce analyst at a large retail company, I own ...
1
vote
1answer
19 views

Robust two-sample test with triplicate measurements?

When testing for a difference in mean between two conditions, biologists typically use a $t$-test, and wring their hands endlessly about how to justify removing outliers. Whereas I typically use a ...
-1
votes
1answer
21 views

Select relevant data to feed model

I am building a regression algorithm to predict the cost of a product defined by several features. I use a supervised rule-based model to do the predictions. The dataset is based on former ...
0
votes
0answers
51 views

Finding outliers in binary data

Say you have data from 10 different sensors about the occurrence of some event - e.g. motion sensors. Each sensor records 1 if they detected the motion sensors and 0 if they didn't. If you have 10 ...
1
vote
0answers
30 views

Outliers and influential observations in elastic net logistic regression

My dataset has many biomarkers and the boxplots of these variables show the presence of many outliers. However, these 'outliers' are real data and not misread observations. I want to use elastic net ...
0
votes
1answer
29 views

Residual vs fitted values (large outlier in predicted value?)

I have attached a plot of residuals vs predicted values for a model i ran. I see 1 large predicted value (extremely large compared to the others) is this a problem for my model? and if so, what can ...
4
votes
0answers
94 views

“Forward search” methods for outlier detection etc. in regression

I am reading Robust Diagnostic Regression Analysis by Anthony Atkinson and Marco Riani. They propose a robust "forward search" method for detecting outliers and other problematic data in regression (...
0
votes
0answers
7 views

How to find out when one of a set of recurrent data is behaving unusually

My problem is finding out a week in which week the data about our sales are "strange". We sell 8 products through a third party vendor V. V sends us, each week, the amount sold for each product. ...
1
vote
1answer
38 views

What are the anomalies/fault/outliers detection algorithms

I'm working on a weather application that uses data coming from multiple sensors in real time (the data is time series), i've made an anomalies detection model using One Class Support Vector Machines, ...
3
votes
0answers
27 views

Using robust regression to detect outliers

Rousseeuw and van Zomeren (1990) propose using robust regression to detect multivariate outliers, particularly in OLS regression. This approach seems to make sense (although I have not studied it in ...
1
vote
1answer
32 views

Omitting certain time periods in VAR

I am using a vector autoregression with a monthly lag, and wish to not include certain months, as they are outliers in my analysis and may distort findings. Is estimating such a VAR (using OLS, then ...
0
votes
0answers
18 views

Algorithm to detect outliers in network sensor messages

I have a network sensor device which generates a number of messages. The message is of format "timeofmessage messagetype messageimportance messagetext". The sensor keeps producing "sensor-ok" messages ...