The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

Filter by
Sorted by
Tagged with
2
votes
0answers
38 views

Decomposition of Mahalanobis distance: Where's my mistake?

Kim (2000) gives a formula for the decomposition of the (squared) Mahalanobis distance for $d$ variables for a random vector $\mathbf{x}$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{...
0
votes
0answers
17 views

Treating Outliers

I have to build a Logistic model to predict churn(telecom). There are large no. of outliers (>1000) in data of 66000 rows. What is the preferred method to treat the outliers? Can it be replaced with ...
0
votes
1answer
22 views

If the datapoint furthest from the y‐axis was excluded, what would happen to the associations seen?

looking at the Figure below, what would happen to the association if the datapoint furtherst from the y-axis was excluded? Current linear association indicates: Diabetes prev = -0.0171183 x BT ...
1
vote
2answers
50 views

Outlier removal, extremes on both ends

A list of numbers and I want to remove the extremes on both ends. The standard deviation is calculated: 26.3 (rounded to 1 decimal) ...
4
votes
1answer
45 views

Simulate multivariate outliers that are hidden in 2D scatterplots

How could I simulate multivariate outliers that are "hidden" in all pairwise 2D scatterplots between the variables? By "hidden" I mean that they can't be seen (as obvious outliers) or detected ...
0
votes
0answers
5 views

Specification on power spectral density for population data

What is the best way to put a specification on the single-side auto-power spectral density (PSD)? We have a product for which we have a time signal. For this signal we calculate the PSD to determine ...
1
vote
2answers
39 views

Is it reasonable to represent mean value without removal of the outliers?

At first, I want to say, I know that mean is outlier sensitive. Problem I have to talk about the lifestyle of the students, where the data is quantitative. Let's say I have to talk, how much time a ...
0
votes
1answer
23 views

Determine significance of an observed value

I have an ECDF of values that do not follow a particular distribution (thought they are slightly normal, they are not). And I wish to determine if a new observed value is significant or an outlier or ...
0
votes
2answers
58 views

Outliers Detection with unlabeled data? [on hold]

I have a dataframe with numeric and categorical variables and no target variables and I need to check for multivariate outliers. Could you suggest a model (using Python) that works good for outliers ...
0
votes
1answer
19 views

Dynamic Outlier Detection in loop with Random Forest regression in R

I'm using randomforest regressor to predict values. I'm trying to make a automatic learning model. My database has let's say 100 rows. So, I train my model with only 10 rows. And one by one I want to ...
1
vote
0answers
25 views

Which average to use to summarise multiple MAPE values?

Suppose you have a model, evaluated using mean absolute percentage error (MAPE), that has made predictions on 1000 different examples. Each example will have an associated MAPE that reflects how well ...
0
votes
0answers
20 views

Does it make sense to validate sales data using RSD?

Here's a nonsense example of data I'm working with: ...
0
votes
0answers
37 views

Extracting the right summary statistics from zero-inflated data sets (i.e. a sparse matrix where everything non-zero is a statistical outlier)

I'm a consumer tech startup founder with rudimentary background in statistics. I need help in processing a large, sparse matrix. I'm logging all actions users are undertaking in my app. I then ...
1
vote
1answer
54 views

How to use PCA to detect outliers?

A PCA will reduce the dimensionality of the original data and construct a subspace generated by eigenvectors of which each represents the (next) highest variance to explain the data. Let's start at ...
0
votes
1answer
68 views

Detecting the outliers from scatter plot

I am trying to understand this question from Everitt et al. (A Handbook of Statistical Analyses Using R) which asks, "Collett (2003) argues that two outliers need to be removed from the plasma data. ...
0
votes
0answers
17 views

Normalization constants in methods for robust outlier detection

A classical rule to detect outliers is to define the interval $[\hat{\mu} - 2 \hat{\sigma}; \hat{\mu} + 2 \hat{\sigma}]$, but it is weell known that the sample mean and standard deviation are not ...
0
votes
1answer
37 views

“Ice skater” / “figure skating” / “ISU” method of discarding outliers

So I need a way of ruling out outliers and "the ice skater method" has been suggested. The person who suggested it has a good deal of experience of doing the task I am doing, so I am certainly ...
0
votes
1answer
47 views

Isolation forest with categorical data?

I understand how isolation forests can work with numeric data, but I wonder how it can work with categorical data? Also, at least when working with Sci-kit-Learn, the recommendation I saw was to ...
0
votes
1answer
30 views

How to exclude events with low data (eg. threshold, outliers)

I have this data set and I want to filter only "Event" with a good conversion rate. We can say that good are those that have a higher than average conversion (but maybe you have better ideas). ...
7
votes
2answers
1k views

Can k-means be used for non normally distributed data?

I read a lot of papers that test k-means with many datasets that are not normally distributed like the iris dataset and get good results. Since, I understand that k-means is for normally distributed ...
2
votes
0answers
38 views

Reference thesis for outliers in likert scale data

I have a data set that was collected in form of a liker scale through questioners. Now, i am trying to eliminate or identify outliers in my data. By far, from all that i have been able to search, ...
0
votes
2answers
616 views

Which would best display the following data if you wanted to display the numbers which are outliers as well as the mean? [closed]

Which would best display the following data if you wanted to display the numbers which are outliers as well as the mean? [4, 1, 3, 10, 18, 12, 9, 4, 15, 16, 32]
6
votes
1answer
47 views

How i add uniformly distributed noisy attributes to data set?

I want to add some artificial outliers to my data set by follow same method below. so, how i can add contaminated data statistically to real data set like Pima Indians Diabetes? info: Pima Indians ...
0
votes
0answers
14 views

how to detect ouliers in skewed data(left skewed distribution) [duplicate]

I have studied a lot of ways of dealing with outliers of normal or multidimensional data. But my problem is about skewed data. How can I find outliers for data with a skewed distribution to the left?
1
vote
0answers
51 views

Outlier Detection via Beta Distribution

Suppose I have a continuous random variable which is bound between 0 and 1. The distribution is left skewed like the picture below: My goal is to identify outliers that are small or farther away from ...
-1
votes
1answer
67 views

Which is the correct method for outlier analysis on a dataset for modelling?

I'm trying to build a regression model but my data-set have many outliers points which I need to analyze and then remove them. There are two ways to do it, 1) First do all the analysis on every ...
0
votes
1answer
35 views

What is the method should I use to calculate similarity in a data set with outliers that must be included?

Information: So I have a data set with 18 vectors with 167 components, each of with has a value with a range of $[-2, 2]$. I am trying to calculate the similarity between one arbitrary vector in ...
2
votes
1answer
70 views

Detecting extrema under uncertainty?

An old version of this question was poorly articulated. Here is another go: I have fifty objects. With a different, independent, unbiased scale for each object, I measure their weights 100 times each ...
2
votes
1answer
24 views

Minimum generalized variance for outlier detection

I'm currently reading the paper Distribution of Variables by Method of Outlier Detection and am trying to understand the section on Minimum generalized variance If ...
0
votes
0answers
18 views

Simulate multivariate outliers

Considering a multivariate linear model $\boldsymbol{Y = XB + E}$, where $\boldsymbol{Y, X, B}$ and $\boldsymbol{E}$ have dimension $n \times m$, $n \times p$, $p \times m$ and $n \times m$, ...
-1
votes
2answers
41 views

In elbow curve how to find the point from where the curve starts to rise? [closed]

I am computing a distance on my data. The result is then being sorted in ascending order. The samples having distance more than a specific threshold are to be marked as outliers and will be discarded. ...
0
votes
1answer
41 views

have new outliers after capping

I'm trying to cap outliers in a column my pandas DataFrame. Here's the boxplot for a column of my original data. boxplot for a column of my original data So, using code from this stackoverflow answer, ...
2
votes
3answers
65 views

What would be the outliers in this specific situation? (given in description)

Recently over a phone interview I was asked this: "I have data consisting of 102 points between 0 and 100 where the first 50 points, starting at 0, are incremented by 0.5 every time. The next two ...
1
vote
1answer
32 views

Outliers of small dataset

I have a python function that takes a list of smaller images boxes (represented as float arrays) and the whole image img in as a ...
0
votes
0answers
65 views

Assessing a binary decission based on continuos and multi-level categorical variables

I have been asked to generate a tool to assess if a particular new set of measurements fit within a list of already accepted ones. The problem is that there are different categorical variables with ...
2
votes
0answers
18 views

Can cross validation be applied to threshold based outlier detection model?

I have a threshold based outlier detection model. I apply PCA then calculate the distance from the centre of the features, and use the MSE to differentiate if the datapoint is a outlier. However, I ...
0
votes
0answers
18 views

What is the relationship between negative autocorrelation and first differenced outliers?

I am currently working with economic data and I am trying to model my dependent variable using several macroeconomic independent variables. When I tried linear regression on the data, I got a high R-...
0
votes
0answers
10 views

Estimation of AO effect based on least squares theory

Given the following model, capturing the presence of an additive outlier in a time series, where $T$ is the time when the additive outlier is recorded and $w_A$ is its effect, $a_t$ is a white-noise ...
0
votes
1answer
32 views

Normality Tests in samples with outliers

I'm making a code in R that contains some parametric and non-parametric tests, like ANOVA and Kruskal-Wallis. To know if I should use one or another I check the "normality" of the test sample. My ...
1
vote
2answers
31 views

scikit-learn IsolationForest no variance feature

I'm using IsolationForest algorithm in order to detect anomalies in my data and to use this model to detect future anomalies in new rows and came across a few questions: Is the model good for ...
0
votes
1answer
32 views

Clarification in Anomaly Detection Algorithm

I am referring to Prof Andrew Ng Coursera ML notes (Week 9). He says that to identify outliers we first model the training data and then fit a Gaussian distribution with probability density $ p(X; \mu ...
0
votes
0answers
18 views

Setting Choosing alpha for generalized extreme Studentized deviate ESD

I'm working on a S-H-ESD implementation and I'm struggling to set the alpha for the ESD. The suggested alpha everywhere is 0.05. Is there a way to calculate an alpha based on the expected percent of ...
1
vote
0answers
27 views

Removing Outliers From Non-Linear Data in the Inappropriate Way Gives a Better Result

I'm doing a cost prediction model for mechanic parts (12000 rows and 43 variables). All the numeric variable(10 numeric variables total) shows no normal distribution. After the log transformation, ...
13
votes
1answer
904 views

Dropping outliers based on “2.5 times the RMSE”

In Kahneman and Deaton (2010)$^\dagger$, the authors write the following: This regression explains 37% of the variance, with a root mean square error (RMSE) of 0.67852. To eliminate outliers and ...
1
vote
1answer
30 views

What outlier score is used here?

I have come across a score function in a program, but I don't exactly understand what it does. This score is a measure of how probable a sampled/created value is. I will describe the procedure for ...
0
votes
1answer
45 views

Coffee ratings: How to do causal inference with high kurtosis/outliers?

I have data (n>1000) on ratings of various coffee features, and then a final overall score. I am interested in inference: What is the effect of a feature on overall score. However, my data has ...
1
vote
1answer
35 views

Is it OK to have only a single class labels in test data for prediction with one-class-svm?

I have a data which has only a single class, namely, '0'. There is no 'not 0' class. The one-class SVM model was trained on a <...
0
votes
1answer
61 views

Should you standardize your variables before or after removing outliers?

Barring the question of how to operationalize outliers, or the utility of doing so, and assuming dependent variables and independent variables are all scaled in the main regression specification (...
0
votes
1answer
45 views

Log-Normalization of skewed data before feeding to neural network models ( autoencoders)

If your input data has few columns that are extremely skewed, It is well known that one would log normalize ( take log and then normalize or standardize) the data before passing to regression ...
0
votes
0answers
63 views

Unsupervised anomaly and outlier detection of database queries

I'm monitoring database queries coming from multiple different applications spread across numerous systems and I'd like to find both anomalous queries as well as outliers in a completely unsupervised ...