Questions tagged [outliers]

An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset. A discomfiting possibility is that these data come from a different population than the one intended to be studied.

Filter by
Sorted by
Tagged with
0 votes
1 answer
17 views

Which cluster analysis method is most robust to outliers and why?

Also I would really appreciate some literature on that topic
user avatar
  • 1
0 votes
0 answers
19 views

Outlier Treatment and Forecasting

I have come across multiple methods regarding outlier treatment: (features = my input/regressor/... matrix) Treat outliers in the entire sample (both features and the variable to be forecasted). ...
user avatar
  • 860
5 votes
2 answers
932 views

Can I remove sample outliers using standard deviation?

I am looking to find find clinical and other measurements to predict a blood metabolite with Elastic-Net Regression models. Can I remove samples with values greater than 1.96 SD from the mean as ...
user avatar
  • 63
1 vote
1 answer
13 views

Should I trim/winsorize raw data or computed metric used in models?

Question: Should I rather winsorise (or trim, where relevant) my raw data, or the intermediary metric I use in my models? Context: My analysis consists in 3 steps: Collect raw data, Compute ...
user avatar
  • 129
1 vote
0 answers
35 views

Bayesian approach to removing outliers from a normal distribution

A lot of what I've seen for Bayesian approaches to removing outliers is for a linear model, not a normal distribution. Is there a way we can take a Bayesian approach to remove outliers from a normal ...
user avatar
0 votes
0 answers
28 views

Modification of Outliers

I have a practical / applied statistics question. I'm dealing with a specialized dataset with a very small sample (i.e. n < 10). In the sequence of observations, it is possible that a new ...
user avatar
0 votes
0 answers
14 views

How to conduct EM algorithm when there are some outliers in GMM Models?

I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models: Suppose that the observed data contains several outliers. The mixture model could be: $$ ...
user avatar
  • 1
0 votes
0 answers
7 views

Testing alternatives for Three-Way Anova that violates normality assumptions (outliers)

I am looking to do inference on Three-Way Anova model, but after looking at the residuals I saw a few violations: non-normality and a few outliers (as seen below). My question would be: How do I deal ...
user avatar
0 votes
0 answers
34 views

Log Transformation to treat outliers

I am trying to replicate a research paper as part of my Applied Econometrics course, and I came across a particularly vague statement in the reference paper. "Following Malmendier and Tate (2005),...
user avatar
0 votes
0 answers
19 views

Smoothing time series with Adjusted R2-weighted averages

I have two parameters (a,b) resulting from an exponential estimation of a curve. I have estimated this curve every hour for one month. In other words, I have a total of 720 parameters a and b, and I ...
user avatar
1 vote
1 answer
33 views

Detecting outliers in a multiple time-series

I have some broker prices incoming in real-time for several products. Sometimes a broker makes a typo and sends a wrong price, or my parsing engine assigns the price to the wrong product - these are ...
user avatar
  • 299
0 votes
0 answers
11 views

The reason behind fixing the IQR coefficient value in Interquartile Range method

To find the outliers, one common approach is to use the Interquartile Range method, especially when you know your data does not follow Gaussian distribution. ...
user avatar
  • 151
0 votes
0 answers
33 views

How to handle regressions with outliers in panel data

I have collected panel data for some countries. I am running several regressions, from simple OLS to IV to panel data methods combined with IV etc. Unfortunately, I see in R that some observations are ...
user avatar
0 votes
0 answers
27 views

Detect and remove outliers from unknown distribution

I have completed a range of steady-state CFD simulations on building roofs. A contour map of the resulting variable is displayed in the Figure below with the corresponding values on the left side. ...
user avatar
0 votes
0 answers
24 views

What is the difference between robustlmm and clubsandwich in R?

Excuse my ignorance, I am trying to get around a problem with my statistics that involves severe outliers issues, with heteroskeskedacity. My model using linear mixed models, in R, with repeated ...
user avatar
0 votes
1 answer
47 views

Do we need to split the data for Unsupervised Anomaly Detection?

I'm struggling with understanding the concept of splitting data for unsupervised anomaly/outlier detection. You can find all approaches here. I found some papers and implementations that didn't split ...
user avatar
  • 385
1 vote
1 answer
27 views

Which raw data to include for heterogenous autoregressive (HAR) model

I constructed the realized variance of bitcoin returns per day from 8-10-2015 to today. The realized variance is calculated by taking the cumulative squared intra-day returns. 5-minute high frequency ...
user avatar
  • 41
0 votes
0 answers
23 views

Ggplot geom_boxplot showing incorrect upper whisker and additional outlier

I am not sure why my boxplot created with ggplot geom_boxplot is showing an incorrect upper whisker and showing the data point (value = 7) as an outlier for "Male" grouping red boxplot. I ...
user avatar
  • 1
0 votes
0 answers
20 views

Sample of Runners - Can the Group Run 2.5 Miles in 20 mins?

I have a dataset where there are 6 runners. Each runner runs as far as they can for 20 mins, and a watcher records their distance (to the nearest 0.1 miles) at certain times, precisely on the minute ...
user avatar
1 vote
1 answer
79 views

Identify outliers in chi-squared goodness of fit test

I am performing a chi-square goodness of fit test to compare an observed value with an expected value. The expected value is calculated from theory. p-value suggests statistical significance. How do I ...
user avatar
  • 11
1 vote
1 answer
49 views

Do I need to transform/standardise my dependent variable?

Attached are the results and the residual plot for my regression of control variables on CEO compensation (TDC1). When I look at the plot my main concerns are the outliers (which I checked to be ...
user avatar
1 vote
1 answer
22 views

Detecting Spikes in a 1-D discrete time series data with unknown underlying distribution

I have a discrete 1-D data set with a value range of 0-100. The underlying distribution is unknown --although we have enough data to fit a model-- to summarize it is a highly right-skewed data set, ...
user avatar
  • 111
0 votes
0 answers
25 views

How to decide which "outliers" to get rid of?

I have thinking about this problem for a while but couldn't quite formulate a proper solution myself. I am also not even sure if it is appropriate to speak of "outliers" or if the term "...
user avatar
-1 votes
1 answer
89 views

Is 6% of your dataset are outliers normal?

My dataset has 80,886 obs and 16 variables. I am using Mahalanobis Distance to detect outliers. And use P-value less than 0.001 as the cut-off. I am getting 5,423 obs as outlier which is 6% of total ...
user avatar
2 votes
1 answer
110 views

Flagging bad time series behavior (Pattern Recognition and Outlier Detection)

I want to get some opinions on how to approach the following problem to do with detecting "unhealthy" behavior in time series data (either using a statistical/analytical model or ML/DL, I do ...
user avatar
  • 47
0 votes
0 answers
31 views

Outlier in grouped data

Existencial crysis here xD. When you want to determine outliers with IQR, and plotting a box-plot what do you plot if your data is structure in the following manner: n-dependent variables (n=6) (...
user avatar
0 votes
0 answers
99 views

Non-parametric outlier extimation

Are there ways to automatically detect outliers ( we can fix uni-dimensional datasets ) when the underlying distribution is difficult to model ? Intuitively, reseampling techniques could help. (1) You ...
user avatar
  • 663
1 vote
0 answers
29 views

Should I have more trees than dimensions for the Isolation Forest?

I have a dataset which has 200 dimensions after pre-processing. I read multiple times that 100 is the recommended number of trees for the Isolation Forest. Since each tree chooses one feature randomly,...
user avatar
0 votes
1 answer
52 views

How to deal with a large number of outliers in biological data?

I´m working on a marine species dataset with R. I would like to compare the biomass and abundance between different sites but I´m not sure how to deal with the large number of outliers. I am aware ...
user avatar
1 vote
0 answers
58 views

Standardized Euclidean Distance over variables distributed as $\chi^2$

I sample $n$ dimension vectors (each sample is a vector). My objective is the detection of outliers. In case those elements would distribute normally, for outlier detection, I could use Standardized ...
user avatar
3 votes
0 answers
29 views

Measuring unusual death [closed]

Given the Prussian Horse Data here: https://www.randomservices.org/random/data/HorseKicks.html Is there a way to find out which corp has an unusually high number of deaths? (Note that Prussian horse ...
user avatar
0 votes
0 answers
30 views

LSTM Autoencoder for online anomaly detection

I would like to use an LSTM-Autoencoder for an anomaly detection task, but in an online setting, meaning we are observing the data as it is streaming in. What I would like to do is given some discrete ...
user avatar
2 votes
2 answers
105 views

Is that possible for a dataset to be 9% outliers?

I have a dataset about solar panels' output power. After visually inspecting the data distribution, I found it is not normal distribution and is a right-skewed distribution with many zeroes. I used ...
user avatar
0 votes
0 answers
27 views

Is it appropriate to fit a linear model to my data?

I have a bunch of outcome/exposure relationships I am trying to fit models to: From these graphs, I am not sure if a simple lm is appropriate. Some of them look ...
user avatar
  • 449
2 votes
4 answers
532 views

Why don't we automatically have outliers when mean and median differ strongly?

Assume you have a data set with information on income of all students in the lecture. The mean value is 1500\$. The median value is however only 800\$. Which of the following conclusions is wrong? The ...
user avatar
1 vote
1 answer
45 views

Outlier management

I apologize in advance for my novice question. I am a part of an interview committee of eight people. We interview 70 applicants for just six positions. All of the applicants are very accomplished. We ...
user avatar
0 votes
0 answers
66 views

Local Outlier Factor (LOF) in Financial Time Series

I'm using Local Outlier Factor (LOF) in financial data with 40 features. When I use the algorithm I can achieve scores outliers, but I can't understand how I can get my algo to tell me the connection ...
user avatar
4 votes
3 answers
1k views

Which are outliers?

I am in the process of solving a Machine Learning challenge, and I want to do it the right way. I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in ...
user avatar
0 votes
0 answers
12 views

Studentized residuals with model averaging

With a standard regression model, we might obtain externally Studentized residuals to identify the worst outliers. But suppose in addition we have lots of possible predictors, all inter-correlated, so ...
user avatar
  • 15.7k
0 votes
0 answers
42 views

How to clean dataset in order to fit to a curve? [duplicate]

I'm trying to fit a dataset to a curve for while, but I'm not managing. The goal is to obtain a curve with equation that fits the data so I can get the parameter x to any value of y. The blue dataset ...
user avatar
0 votes
0 answers
80 views

Detecting and dealing with outliers in a sales prediction dataset of "Rossmann"

I have been working on a dataset for which the task is to forecast the sales of the drug sold by 1115 drug stores of the Rossmann chain. The dataset is fairly large with over 1m records and as many as ...
user avatar
0 votes
1 answer
25 views

Standard deviation estimator without outliers

I have samples that are distributed like this: I want to calculate the standard deviation (or similar) of the main peak without the outliers. Of course I can do this just applying a cut at, say, -5µ. ...
user avatar
0 votes
1 answer
69 views

Why is 50% the best breakdown point for an estimator?

As stated in Wikipedia: Intuitively, we can understand that a breakdown point cannot exceed 50% because if more than half of the observations are contaminated, it is not possible to distinguish ...
user avatar
  • 1
0 votes
0 answers
101 views

Winsorizing or taking the logarithm first?

I testing if I can describe the StockPRice with EPS (=earnings per share), BookValuePS an ESGscore. Before I start I winsorized all my variables. Now I want to take the loagrithm of e.g. BookValuePS ...
user avatar
9 votes
10 answers
3k views

Why is the Median Less Sensitive to Extreme Values Compared to the Mean?

I am sure we have all heard the following argument stated in some way or the other: For a given set of measurements (e.g. heights of students), the mean of these measurements is more "prone"...
user avatar
  • 5,718
2 votes
1 answer
59 views

Comparing outliers in two distributions

I apologize in advance as I am not well-versed in statistics, but I hope that this question makes sense. I have 2 populations which are normally distributed and have a near-identical mean. I would ...
user avatar
1 vote
1 answer
44 views

General Question: Should Legitimate Outliers in the Data be Included or Excluded from Statistical Models?

I have the following (general) question (I know there is no definite answer to this question and it largely depends on the specific data and choice of model): Should Legitimate Outliers in the Data be ...
user avatar
  • 5,718
0 votes
0 answers
41 views

How to improve novelty estimation results when all the training set is important

Edit: as stated here one way to automate searching for the best hyperparameters is using GridSearch functionality. I'm new here so I hope to explain the problem correctly After reading this ...
user avatar
0 votes
1 answer
96 views

How to detect outliers in skewed data?

I have a dataset I need to use to predict the probability of conversion based on the number of days an individual has spent using my app. I got a list of historical users and the number of session ...
user avatar
0 votes
1 answer
259 views

Outliers Logistic Regression

I want to know how to find and remove outliers from my Logistic Regression. I have tried using formula from Faraway, but I don't know is it applicable for logistic regression or not For example my ...
user avatar

1
2 3 4 5
25