Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How are the Cochran's test for inlying variance results interpreted?

I'm running several hypothesis tests to check whether I can assume homocedasticity in some independent data series. The R package outliers [1] offers the function <...
4
votes
1answer
56 views

Trimmed, weighted mean

The trimmed mean (or truncated mean) is a robust version of the mean, designed to be robust to outliers. I am wondering what is the right trimmed version of a weighted average. If I have a sample ...
0
votes
1answer
30 views

Why is One Class SVM predicting that half my dataset consists of outliers?

I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
0
votes
0answers
11 views

Dummy variable in autoregression

I have to estimate an autoregression in a general form: X(t)=a+ bX(t-1) + dummy + e(t). The dummy variable takes 1 for only one observation and zero elsewhere. My question is: Can I estimate the model ...
0
votes
1answer
20 views

Can I remove outliers from a residual plot? Or does this compromise the validity of my model?

I used the function auto.arima to predict sales for the next year. When using only 3 years of the dataset, my results were not good. When I go back 10 years, it improved. However, in order for me to ...
1
vote
0answers
24 views

Dealing with outliers: Interquartile range normalization vs. Winsorization

According to this page -- "When a data set has outliers, variability is often summarized by a statistic called the interquartile range, which is the difference between the first and third ...
0
votes
0answers
13 views

When removing outliers in multiple columns, do I check the columns simultaneously or in succession?

Lets say I have following data: Row X Y 1 11 101 2 12 102 3 13 103 4 14 104 5 15 105 6 40 106 7 16 107 8 17 108 9 18 112.7 Now lets say I would want to remove the outliers that are 1.5xIQR ...
0
votes
0answers
9 views

Out of distribution prediction, not sample

My dataset contains samples that are recorded time signals. I want to predict the start of some event in the time signal. Say I train a neural network with mean squared error as loss function to ...
0
votes
0answers
23 views

Mahalanobis distance to detect multivariate outliers [duplicate]

I have to detect outliers on 3 variables. On the internet I found the mahalanobis distance but I understood I can use it only on multivariate normally distributed data, and my data isn't. So, do you ...
0
votes
0answers
17 views

Outlier detection based only on log probability

I have a problem where I have fit a normalizing flow to data which allows me to calculate the log-likelihood of data points. How should I go about defining an outlier in this situation? Or at least a ...
2
votes
1answer
20 views

Example of a k-dimensional random vector X where each component of the vector is normally distributed but X is not [duplicate]

From the definition of multivariate normal distribution, we know that if a k-dimensional random vector X = (X1, X2, ..., Xk) is (multi-variate) normally distributed if every linear combination of its ...
0
votes
0answers
11 views

Local outlier factor plot

I'm trying to visualize results from LOF algorithm. I came across a type of graph a few times and it makes me question if the calculations I'm doing are correct. The type of graphs I'm talking about ...
0
votes
0answers
13 views

When using multi-variate gaussian distribution modelling for anomaly detection, what are the requirements on the data?

I have been studying Prof. Andrew Ng's ML course. He introduces univariate normal distribution and multivariate normal distribution as ways to model data for anomaly detection. There is a comparison ...
1
vote
0answers
14 views

Defining 'newxreg' argument correctly in the predict() function in R - ARIMA forecasting with tsoutliers [closed]

so I have a dataset Y which I have split into a training and test data set. I have used the tsoutliers function tso() to fit a model to my training data which consists of an AR(3) process plus 15 ...
0
votes
0answers
9 views

Outlier detection in sequential data (operational event logs)

I am currently working on a project where we want to detect outliers in sequential data (operational event logs). The first part of the conceptual method so far is as follows: Collect unlabelled ...
0
votes
0answers
7 views

Correlation: Is there a way to determine whether a particular case is an “influential case” and affecting the correlation coefficient in a large way?

So, for multiple regression, I've read that you don't just remove a case because they are an outlier, but you remove them because they are having undue influence on the model (Field et al., 2012). You ...
1
vote
0answers
32 views

Is this a bad method to detect outliers

I thought of this way of detecting outliers. What are the "bad" properties of this method? For example, say you have a time series, and you want to check if the latest observation is an ...
0
votes
0answers
39 views

R: remove outliers from boxplot or qq-plot?

I am running a regression model, and I need to delete outliers. However, when I ran a boxplot, it asked me to delete at least 100 datas (I only have 700 datas in total). Both y and x variables are ...
0
votes
0answers
16 views

Decision Trees Outliers

How are Decision Trees affected with outliers both regression and classification ones? From my understanding, I see that in the classification context Decision Trees(DT) are robust to outliers as the ...
1
vote
1answer
28 views

how do I Clean the Data when iqr is qual to 0?

I want to clean my data but i faced a variable with more than 86 percent zero and consequently iqr=0 when i want to clean outlier, that feauture are eliminated because all of nonzeros were defined as ...
0
votes
0answers
19 views

Outliers for time series that is often flat?

I have some times series and am attempting some outlier detection. The time series can be considered stationary but with volatility clustering. Hence, I am currently detecting outliers by estimating ...
1
vote
1answer
22 views

Modified distance functions for a cluster analysis

I'm developing some software to allow users to perform various kinds of clustering on some data using a pairwise distance matrix (k-medoids is the main method). I would like to allow the user to tune ...
0
votes
1answer
35 views

What are the statistical drawbacks of my naive and simple approach to discarding outliers?

I'm trying to measure and compare the performance of some computer programs. The simplest idea that comes to mind is to time, say, 10 runs in a row and present the arithmetic mean of the 10 timings as ...
0
votes
0answers
24 views

Outlier as a Measurement of Popularity

disclaimer: I'm not a statistician, analyst, or anything professional in this field and have no formal training. This is just a personal project so apologies if this is a naïve question. Please let me ...
1
vote
1answer
44 views

Is it better to do per-class anomaly detection on P(x, y) or P(x | y)?

(Not an expert in anomaly detection.) I'd like to experiment with per-class anomaly detection. That is, we have a feature vector $x$, and a classifier that predicts its class $\hat{y}$. I'd like to ...
1
vote
0answers
19 views

Addressing an observation which exceeds Cook's Distance

During my data analysis, I've been modelling the effect of 13 predictor variables on one response variable, house prices. When looking into possible transformations for my predictor variables, I was ...
0
votes
0answers
15 views

How can I do one class learning for outlier detection?

I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
0
votes
0answers
18 views

How to clean up outliers in regression which cannot be visualized?

Recently I meet a problem in an interview. Given a dataset $\{(X_i, y_i) \}$ for regression problem, how to detect and clean up outliers before starting using any regression algorithm. The following ...
1
vote
0answers
29 views

Interpreting confidece interval qqPlot of a multiple regression

Hello I have a problem when trying to interpret a multiple linear regression graph and the qqPlot() function. I'm trying to do this with an example data of at least 28.000 obs. I used R to make the ...
0
votes
0answers
35 views

LOOCV to identify outliers?

I am working with a small dataset (15 points), and have built a liner regression model (two explanatory variables). I then performed LOOCV. One of the points has an error that is noticeably higher ...
0
votes
0answers
39 views

Reduce skewness of normalized weights with outliers

I have a vector of values and I want to extract their weights so that they sum up to 1. I currently use this simple formula for normalization: $ w^i = \frac{ x^i }{ \sum {x} }$ The problem is that ...
2
votes
0answers
52 views

Outlier detection in curve fitting

In this picture you can see the measured datapoints (blue) and a curve I fit in (orange). The value with x=10000 seems like an outlier, and I am thinking about removing it, to get a better fitting ...
0
votes
0answers
27 views

What is the best way to detect outliers

I have a dataset of financial data. I want to see if there are any outliers in the dataset, so I make a regression: Stockprice = price_book + price_earnings + price_sales + sector I think a ...
2
votes
0answers
30 views

What is a dragon king?

I am studying a system of cities where the largest city appears to be in many aspects an outlier. The distribution of city size - in any country - are often claimed to follow Zipf's law. According to ...
0
votes
0answers
15 views

Getting the individual dimension behaviour from a multivariate normal model

I am training a multivariate normal model on a set of data (assuming it is the legit case), and then I use this trained model to identify whether a new data point is also legit or a possible outlier ...
2
votes
1answer
46 views

Removing outliers renders a new distribution that has its own outliers

I'm trying to remove all the outliers from a data set. However, after removing them, data points that weren't outliers before are now outliers due to the new distribution. What is the correct ...
0
votes
0answers
13 views

Weighting observations according to their leverage glmnet

I wish to estimate the association between several biological features and a binary health outcome. These biological features, however, have occasional extreme (but valid) outlying observations. Given ...
0
votes
0answers
19 views

Outlier detection using 2D spatial information

I have a list of sensor measurements for air quality with geo-coordinates, and I would like to implement outlier detection. The list of sensors is relatively small (~50). The air quality can gradually ...
2
votes
1answer
43 views

Outlier detection algorithm: Datasets lumped together or separated?

Say, I have two groups of human subjects; groupA and groupB, with a size of nA and nB, respectively. Then, I do the same measurement on both groups (it is actually some results from a CFD simulation); ...
2
votes
1answer
37 views

What stastical modeling techniques I can use to estimate the boundary of data points in an unsupervised way

Suppose I have a dataset (see the explanation of the background later) as shown in the plot below, where each dot is a sample. To human eyes, there is an obvious boundary which separates samples above ...
0
votes
1answer
49 views

How does forecast::tsclean() detect outliers in R?

Does it use a particular z-score? I know that it does apply STL. My data is seasonal, and had quite a few outliers, so I am just wondering how exactly it determined whether a particular data point is ...
0
votes
0answers
162 views

How to Tune Isolation Forest?

Many online blogs talk about using Isolation Forest for anomaly detection. But I got a very poor result. The data used is house prices data from Kaggle. I used IForest and KNN from pyod to identify 1% ...
0
votes
0answers
9 views

What's the good way to find outliers in the continuous physical process measurements time-series? (picture attached) [duplicate]

What's the best way to find outliers in the time-series like the one attached? Just using the knowledge that it is a measurement of a continuous physical process. The data is quite short (200-500 ...
0
votes
1answer
35 views

When running a multiple regression, are both dependent and independent variables scanned for outliers?

I want to run a multiple regression analysis using SPSS. I have used the Mahalanobis d square method to find outliers. However my question is, do I add the dependent variable to the list of ...
0
votes
0answers
28 views

Is Cook's Distance a reliable way to find influential points?

I used Cook's Distance to identify influential points on several datasets and found model performance doesn't improve at all after removing them. So I wonder if Cook's Distance is a reliable way to ...
0
votes
1answer
25 views

Speed Up KNN and Maintaining Accuracy for Anomaly Detection

This question is about using KNN in the context of anomaly detection. If the training dataset is large(10 M data points), KNN will be slow. Is subsampling(i.e. use a small subset of original training ...
0
votes
0answers
28 views

Smoothing train/test data

I am currently working on time series forecasting. I know that the first step is to divide the time series into train and test. Then I also understand that I have to normalize the test set using the ...
0
votes
0answers
31 views

Anomaly Detection for Shifting Patterns

Suppose I build a model(e.g. Mahalanobis distance) to learn the normal pattern shown in green dots of the below plot. Then it is deployed into the production to detect anomalies. But as daily customer ...
0
votes
0answers
46 views

Anomaly/Outlier Detection for Arbitrary Shaped Data?

The following plots use the data from Kaggle's House Prices competition. SalePrice is the target variable. I want to find a method to identify outliers (as shown in red in plot 3 & 4) ...
2
votes
1answer
27 views

What is the best multivariate analysis method to study data with missing values and outliers?

I'm working with a dataset of elemental concentrations in polluted soils. Using the same units, some elements have high values and some have low values. If the concentration of some element is too low ...

1
2 3 4 5
23