Questions tagged [mad]
Median Absolute Deviation (MAD) is a measure of variability in a sample of data, and is often used as an alternative to measures like standard deviation since it is more resistant to outliers. Use the [mae] tag if you are asking about the point forecast accuracy measure called MAD or MAE.
58
questions
0
votes
0
answers
19
views
Comparing Star Ratings
Data Illustration
NOTE: I do not have a stats background. I'm pretty sure I'll use some terminology incorrectly. I have come across several approaches via Python/SQL that I would like to confirm as ...
- 101
1
vote
0
answers
66
views
MAD & Median of weighted GMM
What is the median and median-absolute-deviation of a weighted GMM in terms of component mean and variance? For example, three normal distributions $A$, $B$, $C$ with means $\mu_a,\mu_b,\mu_c$, ...
- 93
0
votes
0
answers
34
views
Constructing a User Profile for Music Taste
My goal is to construct User profiles based on positive (and maybe also negative) interactions with songs. A User has the option to like a song. This would give me a list of likes for each user. With ...
- 31
1
vote
1
answer
113
views
Method for outlier detection in noisy seasonal time series data?
I have around 1000 times series of around 1000 samples, where each sample is 5 minutes a part.
An example of a time series after performing seasonal decomposition is
As we can see the data is very ...
- 151
0
votes
1
answer
43
views
Mean and Mean Absolute Deviation inferential statistics
Consider the following data of runners in a race. The first data is from a certain year the second one is from the year after.
$Last\space year \space|\space Mean: 525s \space|\space MAD: 25s$
$This \...
- 105
1
vote
1
answer
3k
views
Calculating robust z scores with median and MAD
Could someone explain the scaling factors involved in calculating robust z scores using median and MAD please?
As I understand it, conventional Z scores calculated using the mean and SD are sensitive ...
- 13
3
votes
1
answer
2k
views
Median absolute deviation only can be used for anomaly detection for time series without a trend?
I think MAD only can be used to detect anomalies for time series without a trend because it relies only a stable median to detect anomalies. It should be OK for time series with seasonality. Just seek ...
- 867
2
votes
2
answers
1k
views
Replacing outlier removal from IQR to MAD
A common outlier removal formula is Q3 + IQR * 1.5 and Q1 - IQR * 1.5
Outliers can also be removed using Mean Absolute Deviation and Median Absolute Deviation.
Is ...
- 532
1
vote
0
answers
32
views
Check which parameter causes less dispersion in a sample set - When to use CV (Coefficient of Variation) or CD (Coefficient of Dispersion) etc
The overall goal is to extract/engineer features from approximately 100 segments (various lengths but always more than 80 data points) that are as similar as possible to each other and have a very low ...
- 11
1
vote
0
answers
177
views
rolling removing outliers: include or not include
In the paper "Realized kernels in practice: trades and quotes" by O. E.Bandorff-Nielsen etc. cf.
https://onlinelibrary.wiley.com/doi/full/10.1111/j.1368-423X.2008.00275.x
in the section dedicated to ...
- 376
1
vote
0
answers
43
views
Mean Absolute Deviation and data preprocessing
Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Next, based on Mean Absolute Deviation (MAD) we aim to decide if the last point $x_{n+1}$ is outlier or not.
First, let us compute the MAD:
...
- 376
0
votes
0
answers
469
views
Convert median absolute deviation (MAD) to SD for log-normal distribution
How do I convert the $\text{MAD}$ (median absolute deviation from the median) of data that is drawn from a log-normal distribution to the standard deviation of a log-normal distribution? To clarify, ...
- 41
1
vote
3
answers
1k
views
What's the best method to measure relative variability for non normal data?
I have a set of data that states the volume required each month for the next 12 for 750 raw materials.
I would like to determine the variability in the demand for each material, and categorise the ...
- 11
1
vote
1
answer
91
views
Inference using robust statistics
I'm simulating the overall usage of a cluster using historic deployment data.
Due to the nature of the simulation, there are some heavy points (i.e. very low overall usage). As a result, the variance ...
- 119
0
votes
1
answer
317
views
Welch t-test using the median and MAD
Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers.
I'm using this to test ...
1
vote
0
answers
1k
views
Median absolute deviation outlier detection for new data points
I have been reading this article for outlier detection and the method proposed works really well when applied over my data set.
However, it would be great if i could run outlier check for new data ...
- 343
5
votes
1
answer
4k
views
How to estimate the scale factor for MAD for a non-normal distribution?
I understand that the scale factor for normally distributed data is 1.4826 to convert it to a pseudo standard deviation like quantity which could be used with the median for determining confidence ...
- 183
2
votes
1
answer
853
views
MAD & Standard Deviation
Am I correct in saying that you can only use the formula for approximating the standard deviation from MAD, i.e
$$\text{SD } = K \times \text{ MAD }$$
if you know the actual probability ...
3
votes
1
answer
473
views
How to prove 'For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution'
For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution
I came across this statement in the wikipedia page for MAD(Median Absolute Deviation)
but ...
user195689
2
votes
0
answers
140
views
Robust Regression
The Median Absolute Deviation (MAD) is often used as a measure of dispersion of the residuals in robust regression. For univariate data, MAD is computed by first finding the median of the data, then ...
- 21
4
votes
0
answers
504
views
Interpretation of the least absolute deviations linear regression coefficient
In linear regression of $y$ onto $x$, one finds a $\beta_0$ and $\beta_1$ minimizing $\sum \|y - (\beta_1 x + \beta_0)\|^2$. One can show that
$$\beta_1 = \rho(x,y) \frac{\sigma(y)}{\sigma(x)},$$
...
- 181
0
votes
1
answer
138
views
Only four values per site - how do I detect outliers (with MAD)?
I have four weights. I want to test if there is an outlier within those
four weights.
To me, the number seems quite small to detect outliers.
I thought that the Median Absolute Deviation (MAD) ...
- 91
4
votes
1
answer
614
views
How to calculate the median absolute deviation for a continuous function?
I know that the median value $(\text{Med})$ for a variable $X$ characterised by the contentious PDF function $f_X(x)$ can be calculated by finding $\text{Med}$ in:
$\int^\text{Med}_{-\infty} dF_X(x) =...
- 684
1
vote
1
answer
210
views
How does one calculate Fisher-consistency factor for Rousseeuw and Croux's $S_n$ for empirical distribution?
In "Alternatives to the Median Absolute Deviation" (Rousseeuw and Croux, J. Amer. Statistical Assoc, 88(424), 1993, pp.1273–1283), the authors described an estimator of SD better than median absolute ...
- 549
5
votes
3
answers
3k
views
How to normalize if MAD equals zero?
A known way to normalize our feature vectors is:
$$\frac{x_i - \operatorname{median}( X^{(j)})}{\operatorname{MAD}^{(j)}},$$
where $\operatorname{MAD}^{(j)}$ is the median absolute deviation of ...
- 195
7
votes
1
answer
3k
views
Can Median Absolute Deviation (MAD)/SD be used to determine if a distribution is normal or not?
I have recently come across this post on Median Absolute Deviation (MAD). The Wikipedia article here, by the article as an estimator Standard deviation of the distribution is 'k' times MAD, where the ...
- 198
0
votes
0
answers
60
views
Square of MAD as variance equivalent
I have to estimate the error of a 1D array. The usual way of doing this to take the SD of it and square it.
The issue is the following:
The values in the array represent the time series of the ...
- 175
1
vote
1
answer
443
views
Why don't dispersions like median deviation and mode deviation exists on the lines of mean deviation?
Or to say in the formula MD=$\frac{1}{N}\sum_{i=1}^{N}|x_i-\overline{x}|$ why can't we have median or mode instead of mean($\overline{x}$) and as a result speak about additional possible measures of ...
- 351
12
votes
4
answers
3k
views
Is there a version of the correlation coefficient that is less-sensitive to outliers?
The correlation coefficient is:
$$ r = \frac{\sum_k \frac{(x_k - \bar{x}) (y_k - \bar{y_k})}{s_x s_y}}{n-1} $$
The sample mean and the sample standard deviation are sensitive to outliers.
As well, ...
- 481
1
vote
1
answer
210
views
What is the best way to determine which proteins are significantly bound on a testing chip?
I've got a question about the data from a biological experiment.
Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
- 13
0
votes
0
answers
2k
views
What should I use to establish a threshold to detect threshold crossings?
I have data graphed in green here:
I want to detect the peaks but I don't know how to define the threshold values. The blue lines are mean ± standard deviation. Using that, I would miss the three ...
- 101
3
votes
0
answers
597
views
alternative to IQR [duplicate]
I am currently making a short literature study of robust and efficient estimators. Some very well known are the median absolute deviation (MAD) and the interquartile range (IQR). However they both ...
- 131
4
votes
0
answers
7k
views
When the Median Absolute Deviation (MAD) is zero
Suppose my data look like the following:
(10, 10, 10, 10, 10, 0)
Would it be possible to remove an outlier in this distribution using the median absolute deviation?
Of course, you wouldn't need to ...
- 49
0
votes
1
answer
71
views
Reducing density of outliers
I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days)
Now the TTB values less than 14 or so are possible. But not with ...
10
votes
1
answer
34k
views
Determine outliers using IQR or standard deviation?
Similar to ... which doesn't have the answer I'm looking for.
The data set has a normal distribution.
For the project I'm working on, outliers have to be determined over residuals of breeding data....
- 223
5
votes
1
answer
1k
views
Median absolute deviation: impact/bias when forcing a median of zero
I want to estimate the median absolute deviation (MAD) of a signal. The MAD is defined as the median of the absolute difference between the signal and its median.
Now I have a signal that I know for ...
- 147
16
votes
2
answers
6k
views
Median absolute deviation (MAD) and SD of different distributions
For normally distributed data, the standard deviation $\sigma$ and the median absolute deviation $\text{MAD}$ are related by:
$\sigma=\Phi^{-1}(3/4)\cdot \text{MAD}\approx1.4826\cdot\text{MAD},$
...
- 161
14
votes
1
answer
374
views
Are there non-trivial settings where the MAD statistic has a closed-form density?
The MAD statistic of an iid sample $(x_1,\ldots,x_n)$ is defined as the median of the absolute deviation from the median:
$$
\text{mad}(x_1,\ldots,x_n)=\text{med}\left\{|x_i-\text{med}(x_1,\ldots,x_n)|...
- 94.5k
15
votes
2
answers
12k
views
MAD formula for outlier detection
Does anyone know what is the name of this formula?
$$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$
where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
- 153
1
vote
1
answer
111
views
Better than Mad, Derive $Q_n$
Could someone be so kind as to walk me through how to do this equation on a dataset?
on pg 5 (marked as 1277) equation 3.3 of this paper.
$$Q_n = c|x_i-x_j|_{(k)},\;\;1\leq i<j\leq n,k\approx{n\...
- 157
0
votes
0
answers
1k
views
Bar plot with median +/- median absolute deviation in scientific publication
I know bar plots usually represents mean +/- sem, but if the data is non-normal, is it ok to bar plot median +/- mad?
I don't like box plots.
Suggestison?
- 49
1
vote
0
answers
212
views
What's the proper way to do automatic iterative outlier rejection?
I know this is a touchy subject, so I'll proceed with caution...
I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
- 2,392
1
vote
0
answers
190
views
Proving the consistancy of the MAD
I am trying to prove that the median absolute deviation from the median (MAD), with k=1.4862, is a consistent estimator of the standard deviation.
1
vote
1
answer
229
views
Alternatives to MAD to find a yardstick to assess data
In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is:
$$
Sn = C\, {\...
- 19
12
votes
1
answer
1k
views
What distribution has the maximum entropy for a known mean absolute deviation?
I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum ...
- 223
2
votes
1
answer
2k
views
Understanding MAD Results
I'm using R to calculate the median absolute deviation for a few distributions, but some of the values I'm calculating do not seem realistic at all. I have the following distribution:
...
- 244
5
votes
2
answers
5k
views
MAD in relation to 95% confidence
MAD (Median Absolute Deviation) is:
$\text{MAD} = M_i(|x_i-M_j(x_j)|)$
where $M()$ is the median operator ($M_i(x_i) = \text{median}(x_1,...,x_n)$).
I'd like to scale the MAD in such a way as to ...
- 244
14
votes
1
answer
2k
views
Does a median-unbiased estimator minimize mean absolute deviance?
This is a follow-up but also a different question of my previous one.
I read on Wikipedia that "A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as ...
- 393
4
votes
0
answers
1k
views
How to test stability/variability of statistic over time, using R
I have a long time series data (running several years). I have split the data into several series, each of 1 month duration.
I then perform some custom calculations on each of the 'shortened' time ...
8
votes
2
answers
3k
views
Median + MAD for skewed data
I am trying to figure out what happens if you apply Hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of Hampel's method over z-...
- 631