All Questions
Tagged with outliers normal-distribution
42 questions
0
votes
1
answer
75
views
Hypothesis testing - Newbie blockers - Update and more
Brief : I'm from manufacturing industry, a processing machine in our production line used to do pressing, polishing and QA one after the other. Now we have a new machine that will perform these at the ...
- 3
4
votes
2
answers
1k
views
Is calculating skewness necessary before using the z-score to find outliers?
For example, if I specify a z-value of 3, then I would look at both sides and know its position in the distribution (99.73%).
Would this change if I have a left or right skewed distribution? Would I ...
- 125
5
votes
1
answer
3k
views
How to define the line to fit in Q-Q plot?
I'm trying to figure out if my data follows a normal distribution and if it contains outliers. I have plotted the histogram and now I would like to plot the quantile-quantile (Q-Q) plot. My point is, ...
- 153
1
vote
1
answer
4k
views
Does IQR method for outliers work for non-normal data?
Any observations that are more than 1.5 IQR below Q1 or more than 1.5 IQR above Q3 are considered outliers. However does this theory still hold when a data set is not normally distributed?
Outlier ...
- 113
3
votes
4
answers
815
views
Why don't we automatically have outliers when mean and median differ strongly?
Assume you have a data set with information on income of all students in the lecture. The mean value is 1500\$. The median value is however only 800\$. Which of the following conclusions is wrong?
The ...
- 111
2
votes
1
answer
397
views
Comparing outliers in two distributions
I apologize in advance as I am not well-versed in statistics, but I hope that this question makes sense.
I have 2 populations which are normally distributed and have a near-identical mean. I would ...
4
votes
2
answers
2k
views
Does classic outlier detection assume normality?
My classmate told me he was showing his work in some stuff statistics-based and some time he was showing a boxplot and using it as outlier detection then his professor said 'it's not even correct, the ...
- 1,230
2
votes
1
answer
431
views
Example of a k-dimensional random vector X where each component of the vector is normally distributed but X is not [duplicate]
From the definition of multivariate normal distribution, we know that if a k-dimensional random vector X = (X1, X2, ..., Xk) is (multi-variate) normally distributed if every linear combination of its ...
- 123
0
votes
0
answers
401
views
Generating Multivariate Outliers in R
I have generated a data-set following the Gaussian distribution with 5 dimensions. The data-set is 1Million record and I am seeking planting 20% outliers, is there a more neat way in R to generate ...
- 1
10
votes
2
answers
4k
views
Can k-means be used for non normally distributed data?
I read a lot of papers that test k-means with many datasets that are not normally distributed like the iris dataset and get good results. Since, I understand that k-means is for normally distributed ...
- 195
6
votes
1
answer
66
views
How i add uniformly distributed noisy attributes to data set?
I want to add some artificial outliers to my data set by follow same method below.
so, how i can add contaminated data statistically to real data set like Pima Indians Diabetes?
info:
Pima Indians ...
- 195
1
vote
1
answer
3k
views
Normality test and Outlier detection [duplicate]
In this question, I would like to ask two things:
outlier detection
normality test
Details are as follows:
I need to detect and remove outliers in my data. Before doing that, I want to test if my ...
1
vote
1
answer
2k
views
How to deal with the normally distributed data with outliers?
My question might sound strange but this is the situation I'm dealing with!
I have a dataset, consisting of 4 data series, each a measurement of a parameter of a biological sample. We have 31 samples. ...
- 43
1
vote
1
answer
2k
views
Anomaly detection on 1D data with multiple gaussian distributions
My core problem is to set a cutoff to my one dimension data between normal with abnormal. I think this is a 'anomaly detection' problem.
My Data
My data is one dimension, consists with below:
(...
- 11
11
votes
2
answers
8k
views
Decision trees, Gradient boosting and normality of predictors
I have a question regarding the normality of predictors. I have 100,000 observations in my data. The problem I am analysing is a classification problem so 5% of the data is assigned to class 1, 95,000 ...
- 365
0
votes
1
answer
1k
views
How to use Tukey's Biweight Function to appropriately weight outliers to generate a normal distribution
I am working with a distribution that has outliers beyond 1.5*3rd Qu.. I'm using Shankar, et al. Recommendations for the validation of immunoassays used for ...
- 215
0
votes
1
answer
86
views
Linear regresssion on body mass index
I am using a continuous variable of body mass index. I checked the distribution using statistical tests and determined it is not normally distributed. I think these results are driven by outliers, ...
1
vote
0
answers
371
views
Bayesian inference for outliers detection
Can you please comment, validate, correct my reasoning here?
I want to identify outliers of the value to mass ratio (V/M) of rice, just to give a simple univariate example.
Prior information: rice ...
- 165
2
votes
0
answers
7k
views
Outlier detection: Normal distribution
I was reading the following link:
http://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
In the link they have made the following statement:
Outliers are extreme values that fall ...
- 223
4
votes
1
answer
4k
views
The algorithm behind pcl::StatisticalOutlierRemoval
I am investigating how a particular statistical outlier removal algorithm of 3D data works, but I am not able to properly figure out what they are doing.
The purpose of this algorithm, is to removal ...
- 1,685
0
votes
0
answers
57
views
For univariate outlier analysis should I use z score if my data is skewed? [duplicate]
If my data is skewed, does it mean that my data does not follow a normal distribution? How do we define various distributions?
What type of outlier analysis do I perform for the different type of ...
- 133
1
vote
0
answers
880
views
What is the influence of outliers in discriminant analysis?
In discriminant analysis like a Canonical Regression Analysis, the outliers tend to skew the results of the analysis, but nowhere can I find "how much" the outliers influences the analysis.
What I'd ...
- 11
3
votes
1
answer
6k
views
Do I remove outliers to make the data normally distributed for ANOVA? [duplicate]
I am trying to conduct a 3 x 2 mixed ANOVA (my measure is reaction time data) and my data is currently not normally distributed. I have read that ANOVA is robust to the violations of normality so I ...
- 31
5
votes
0
answers
4k
views
Detecting outliers in percentages
My dataset looks like below -
Total Success Percentage
100 65 65%
50 25 50%
30 20 66.6%
50 40 80%
Plot -
Each row is ...
- 51
1
vote
2
answers
1k
views
Should I remove the outlier?
I want to run an ANOVA test. I am therefore testing for normality. I have tested each group and the residuals (group together)for normality. My data sample does not look approximately normal. However ...
user123867
-1
votes
1
answer
239
views
Test for normality with outliers produces strange p-values
I try to create some example that show how an outlier causes non-normality. Therefore I created two datasets:
A dataset with normal distributed data
...
- 113
0
votes
1
answer
4k
views
Finding outlier values for non-normally distributed data
I have univariate data (38 is the sample size).The distribution is certainly not normal. How can I find the outliers? I used z-score but am not getting a desired result.
- 13
2
votes
0
answers
1k
views
Deriving mean and variance of the posterior distribution
I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
- 53
0
votes
2
answers
253
views
How to show that a dataset does not contain significant outliers?
I have largish dataset: there is 200 variables and 100 samples. How could I show that the dataset does not contain any significant outliers? All variables have the same unit (millimeters) and have ...
0
votes
0
answers
2k
views
Finding a confidence interval for observations being outliers
Suppose I have a sample with sample size $N$ that is obtained experimentally, e.g. I have counted the number of birds at a certain location at a certain time.
Now suppose that the sample (the number ...
- 1
3
votes
1
answer
3k
views
Anomaly detection: multivariate Gaussian distribution
I am trying to do anomaly detection on a heterogeneous dataset (There are unknown groups present in the dataset). I want to try multivariate Gaussian distribution based approach, but I was thinking of ...
- 251
2
votes
1
answer
66
views
Analyzing regression results
I have done a regression model where i determine the number of cubes (independent variable) based on the amount of units i started with for each product type (dependent variables, ...
- 1,091
2
votes
0
answers
710
views
Can Chauvenet's criterion be used with non-normal data?
Can I use Chauvenet's criterion on set of observations where a normal distribution cannot be assumed?
- 163
2
votes
0
answers
3k
views
qqPlot with Two Outliers
If I have a QQ plot with two extreme outliers (picture below) how should I interpret it? Do I drop the outliers? Can I treat it as normal?
- 121
1
vote
1
answer
265
views
Alternatives to MAD to find a yardstick to assess data
In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is:
$$
Sn = C\, {\...
- 19
2
votes
1
answer
965
views
Problem with identifying outliers
I'm running a biological experiment with rodents, have two groups (each consists of 26 animals), where one is treated with a chemical, and one is control (saline).
In one variable, there doesn't seem ...
- 81
17
votes
3
answers
7k
views
Estimating parameters of a normal distribution: median instead of mean?
The common approach for estimating the parameters of a normal distribution is to use the mean and the sample standard deviation / variance.
However, if there are some outliers, the median and the ...
- 3,398
7
votes
3
answers
7k
views
How to estimate the parameters of a Gaussian distribution sample with outliers?
I have a sample of length $N$, mostly taken from a Gaussian distribution with unknown mean and variance. Of those $N$ samples, some proportion of them (typically less than 1-2%) are outliers, taken ...
- 171
3
votes
2
answers
2k
views
Dixon test and normal distribution
I hope Michael Chernick will read this question.
I have applied Dixon test to 100k rows. Rows are like-
1 1.819691 2.565696 3.317881 1.491987 ...
- 227
6
votes
2
answers
5k
views
Outlier removal prior to mixed-effect modelling
I'm analysing reaction time data from a grammaticality judgement task (collected in a masked-priming experiment). The stimulus were noun-noun compounds, including 3 types of compounds (depending on ...
- 107
7
votes
4
answers
6k
views
Detect outliers in mixture of Gaussians
I have a ton of univariate samples ($x_i \in \mathbb{R}^+$). I'd like an automated method to check for outliers and identify the outliers, if any are present. A reasonable model for the distribution ...
- 6,738
8
votes
1
answer
255
views
Is there a name for the high sensitivity of frequency of extreme data points to the mean of a normal distribution?
I remember hearing an argument that if a certain population of people has a mean IQ of 110 rather than the typical 100, it will have far more people of IQ 150 than a similarly-sized group from the ...
- 769