Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

Why is One Class SVM predicting that half my dataset consists of outliers?

I am currently working on a dataset with 14 continuous features, a categorical target over five classes, and 90,000 samples. My current goal is to explore outliers in the dataset, and to that end I ...
0
votes
0answers
8 views

Dummy variable in autoregression

I have to estimate an autoregression in a general form: X(t)=a+ bX(t-1) + dummy + e(t). The dummy variable takes 1 for only one observation and zero elsewhere. My question is: Can I estimate the model ...
0
votes
1answer
20 views

Can I remove outliers from a residual plot? Or does this compromise the validity of my model?

I used the function auto.arima to predict sales for the next year. When using only 3 years of the dataset, my results were not good. When I go back 10 years, it improved. However, in order for me to ...
1
vote
0answers
19 views

Dealing with outliers: Interquartile range normalization vs. Winsorization

According to this page -- "When a data set has outliers, variability is often summarized by a statistic called the interquartile range, which is the difference between the first and third ...
1
vote
1answer
1k views

Unsupervised Outliers detection on time series

So I am looking ways to improve my current implementation of detecting outliers in work schedule. My data set is badge swipes for people. The current implementation finds outliers on in-times and out-...
0
votes
0answers
13 views

When removing outliers in multiple columns, do I check the columns simultaneously or in succession?

Lets say I have following data: Row X Y 1 11 101 2 12 102 3 13 103 4 14 104 5 15 105 6 40 106 7 16 107 8 17 108 9 18 112.7 Now lets say I would want to remove the outliers that are 1.5xIQR ...
0
votes
0answers
8 views

Out of distribution prediction, not sample

My dataset contains samples that are recorded time signals. I want to predict the start of some event in the time signal. Say I train a neural network with mean squared error as loss function to ...
34
votes
1answer
2k views

Link Anomaly Detection in Temporal Network

I came across this paper that uses link anomaly detection to predict trending topics, and I found it incredibly intriguing: The paper is "Discovering Emerging Topics in Social Streams via Link Anomaly ...
0
votes
0answers
22 views

Mahalanobis distance to detect multivariate outliers [duplicate]

I have to detect outliers on 3 variables. On the internet I found the mahalanobis distance but I understood I can use it only on multivariate normally distributed data, and my data isn't. So, do you ...
0
votes
0answers
17 views

Outlier detection based only on log probability

I have a problem where I have fit a normalizing flow to data which allows me to calculate the log-likelihood of data points. How should I go about defining an outlier in this situation? Or at least a ...
2
votes
0answers
30 views

Name for spurious linear Regression Plots [duplicate]

Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
2
votes
1answer
19 views

Example of a k-dimensional random vector X where each component of the vector is normally distributed but X is not [duplicate]

From the definition of multivariate normal distribution, we know that if a k-dimensional random vector X = (X1, X2, ..., Xk) is (multi-variate) normally distributed if every linear combination of its ...
0
votes
0answers
11 views

Local outlier factor plot

I'm trying to visualize results from LOF algorithm. I came across a type of graph a few times and it makes me question if the calculations I'm doing are correct. The type of graphs I'm talking about ...
0
votes
0answers
13 views

When using multi-variate gaussian distribution modelling for anomaly detection, what are the requirements on the data?

I have been studying Prof. Andrew Ng's ML course. He introduces univariate normal distribution and multivariate normal distribution as ways to model data for anomaly detection. There is a comparison ...
1
vote
0answers
14 views

Defining 'newxreg' argument correctly in the predict() function in R - ARIMA forecasting with tsoutliers [closed]

so I have a dataset Y which I have split into a training and test data set. I have used the tsoutliers function tso() to fit a model to my training data which consists of an AR(3) process plus 15 ...
1
vote
1answer
290 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
3
votes
1answer
194 views

Von Mises distribution to detect outliers

I am working out the difference between two angles from a circle, and I work out the mean difference across 96 trials in 10 separate samples. In order to detect outliers for statistical analysis, ...
0
votes
1answer
341 views

Using Standardized Data or Normal Data With Outliers Excluded

I'm currently working with a large multivariate data set where I plan to use K-Means to try to find any associations in the data. I'm not particularly well-versed when it comes to statistics, though ...
1
vote
2answers
755 views

Dealing with probable outliers in the dependent variable

I am trying to fit a simple regression to a data set with ~45,000 observations. The dependent variable is revenue growth, but I'm concerned some observed values are incorrectly entered data. To ...
0
votes
0answers
9 views

Outlier detection in sequential data (operational event logs)

I am currently working on a project where we want to detect outliers in sequential data (operational event logs). The first part of the conceptual method so far is as follows: Collect unlabelled ...
5
votes
2answers
2k views

Should I remove any out-liers before splitting the data?

I've split my data into three sets before doing any pre-processing; training, validation and testing. I thought that any pre-processing tasks have to take place after splitting the data. However, some ...
0
votes
0answers
7 views

Correlation: Is there a way to determine whether a particular case is an “influential case” and affecting the correlation coefficient in a large way?

So, for multiple regression, I've read that you don't just remove a case because they are an outlier, but you remove them because they are having undue influence on the model (Field et al., 2012). You ...
1
vote
1answer
447 views

Support vector regression with one large outlier

I have a question regarding support vector regression, best summarized by the chart below on simulated data of a linear function with a bit of noise. In essence, why does increasing epsilon rotate the ...
1
vote
1answer
22 views

Modified distance functions for a cluster analysis

I'm developing some software to allow users to perform various kinds of clustering on some data using a pairwise distance matrix (k-medoids is the main method). I would like to allow the user to tune ...
1
vote
1answer
28 views

how do I Clean the Data when iqr is qual to 0?

I want to clean my data but i faced a variable with more than 86 percent zero and consequently iqr=0 when i want to clean outlier, that feauture are eliminated because all of nonzeros were defined as ...
0
votes
1answer
35 views

When running a multiple regression, are both dependent and independent variables scanned for outliers?

I want to run a multiple regression analysis using SPSS. I have used the Mahalanobis d square method to find outliers. However my question is, do I add the dependent variable to the list of ...
1
vote
0answers
32 views

Is this a bad method to detect outliers

I thought of this way of detecting outliers. What are the "bad" properties of this method? For example, say you have a time series, and you want to check if the latest observation is an ...
0
votes
0answers
39 views

R: remove outliers from boxplot or qq-plot?

I am running a regression model, and I need to delete outliers. However, when I ran a boxplot, it asked me to delete at least 100 datas (I only have 700 datas in total). Both y and x variables are ...
0
votes
0answers
15 views

Decision Trees Outliers

How are Decision Trees affected with outliers both regression and classification ones? From my understanding, I see that in the classification context Decision Trees(DT) are robust to outliers as the ...
1
vote
1answer
44 views

Is it better to do per-class anomaly detection on P(x, y) or P(x | y)?

(Not an expert in anomaly detection.) I'd like to experiment with per-class anomaly detection. That is, we have a feature vector $x$, and a classifier that predicts its class $\hat{y}$. I'd like to ...
5
votes
3answers
3k views

Are deep neural networks robust to outliers?

Tree based models such as (gradient boosting or random forest) have a lot of advantages, such as robust to collinearity and outliers. I can see deep neural networks (MLP) are robust to collinearity. ...
1
vote
0answers
69 views

Spearman-like correlation when X and Y are interchangeable?

I have some data consisting of pairs of brothers, and I want to look at correlation within the pairs. There is no logical basis for determining which one will be "X" and which one will be "Y". So, ...
0
votes
0answers
19 views

Outliers for time series that is often flat?

I have some times series and am attempting some outlier detection. The time series can be considered stationary but with volatility clustering. Hence, I am currently detecting outliers by estimating ...
2
votes
1answer
252 views

How to predict when an outlier is going to occur if it's shown to occur repeatedly?

I know my title seems to go against what an outlier is, but I don't know any other way of phrasing it. Let's say I have a spreadsheet, and a column with the following values: [From newest-to-oldest] ...
0
votes
1answer
35 views

What are the statistical drawbacks of my naive and simple approach to discarding outliers?

I'm trying to measure and compare the performance of some computer programs. The simplest idea that comes to mind is to time, say, 10 runs in a row and present the arithmetic mean of the 10 timings as ...
0
votes
0answers
24 views

Outlier as a Measurement of Popularity

disclaimer: I'm not a statistician, analyst, or anything professional in this field and have no formal training. This is just a personal project so apologies if this is a naïve question. Please let me ...
0
votes
1answer
31 views

How to find categorical contributing factors for an anomaly?

Given a house sales dataset with number of houses sold each day and their attributes (i.e., price, number of rooms, size, etc.) - if on a specific day there's a spike/drop in sales, what are some ...
2
votes
2answers
126 views

Regression without a dependent variable

I want to construct a multivariate model to find outliers in the data. The data I have is similar to the iris data (without the Species data attribute, I only have the first 4 attributes) ...
0
votes
2answers
233 views

Outliers Detection with unlabeled data? [closed]

I have a dataframe with numeric and categorical variables and no target variables and I need to check for multivariate outliers. Could you suggest a model (using Python) that works good for outliers ...
4
votes
1answer
482 views

Changing sensitivity (cval) in tsoutliers resulting in unexpected results

I am using the excellent tsoutliers R package to detect outliers (additive outliers, temporary changes etc.), but the cval ...
1
vote
0answers
19 views

Addressing an observation which exceeds Cook's Distance

During my data analysis, I've been modelling the effect of 13 predictor variables on one response variable, house prices. When looking into possible transformations for my predictor variables, I was ...
0
votes
0answers
14 views

How can I do one class learning for outlier detection?

I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
0
votes
0answers
18 views

How to clean up outliers in regression which cannot be visualized?

Recently I meet a problem in an interview. Given a dataset $\{(X_i, y_i) \}$ for regression problem, how to detect and clean up outliers before starting using any regression algorithm. The following ...
1
vote
0answers
29 views

Interpreting confidece interval qqPlot of a multiple regression

Hello I have a problem when trying to interpret a multiple linear regression graph and the qqPlot() function. I'm trying to do this with an example data of at least 28.000 obs. I used R to make the ...
1
vote
2answers
343 views

Detecting misclassified data points using distance matrix

I have a data set of around 65,000 items from a public database that have each been assigned to one of around 1,800 classes, but unfortunately I am aware that quite a few are misclassified. My goals ...
0
votes
0answers
35 views

LOOCV to identify outliers?

I am working with a small dataset (15 points), and have built a liner regression model (two explanatory variables). I then performed LOOCV. One of the points has an error that is noticeably higher ...
2
votes
2answers
1k views

Finding outliers in multiple dimensions

I'm working on dataset which isn't normally distributed. It contains three dimensions: cost, discount and profit. I'm trying to find outliers in all these dimensions. I used $\text{z-score}$ to find ...
0
votes
1answer
90 views

Big outlier in dependent variable

I have my data from the official statistics office of my country and I rechecked multiple times already. I have a big outlier skewing all my glm (poisson) modells to the extreme (like 5 times the ...
0
votes
2answers
333 views

How to downweigh outlier in a sum?

I have a simple problem. Assume following dataset: resids <- c(,9,8,7,12,14,8,9,15,4,9,10,200) n <- length(resids) p <- 2 Using this dataset I want to ...
0
votes
0answers
37 views

Reduce skewness of normalized weights with outliers

I have a vector of values and I want to extract their weights so that they sum up to 1. I currently use this simple formula for normalization: $ w^i = \frac{ x^i }{ \sum {x} }$ The problem is that ...

1
2 3 4 5
23