Questions tagged [mad]
Median Absolute Deviation (MAD) is a measure of variability in a sample of data, and is often used as an alternative to measures like standard deviation since it is more resistant to outliers. Use [tag:mae] if you are asking about the point forecast accuracy measure called MAD or MAE.
47
questions
0
votes
0answers
27 views
Convert median absolute deviation (MAD) to SD for log-normal distribution
How do I convert the $\text{MAD}$ (median absolute deviation from the median) of data that is drawn from a log-normal distribution to the standard deviation of a log-normal distribution? To clarify, ...
1
vote
3answers
110 views
What's the best method to measure relative variability for non normal data?
I have a set of data that states the volume required each month for the next 12 for 750 raw materials.
I would like to determine the variability in the demand for each material, and categorise the ...
1
vote
1answer
54 views
Inference using robust statistics
I'm simulating the overall usage of a cluster using historic deployment data.
Due to the nature of the simulation, there are some heavy points (i.e. very low overall usage). As a result, the variance ...
0
votes
0answers
117 views
Modified Z score of a sample
I would like advice on how to calculate a modified Z score for multiple samples drawn from a population. I understand that when working with standard Z score the standard error of the mean woudl be ...
0
votes
1answer
78 views
Welch t-test using the median and MAD
Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers.
I'm using this to test ...
1
vote
0answers
442 views
Median absolute deviation outlier detection for new data points
I have been reading this article for outlier detection and the method proposed works really well when applied over my data set.
However, it would be great if i could run outlier check for new data ...
1
vote
1answer
272 views
MAD & Standard Deviation
Am I correct in saying that you can only use the formula for approximating the standard deviation from MAD, i.e
$$\text{SD } = K \times \text{ MAD }$$
if you know the actual probability ...
2
votes
1answer
165 views
How to prove 'For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution'
For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution
I came across this statement in the wikipedia page for MAD(Median Absolute Deviation)
but ...
2
votes
0answers
77 views
Robust Regression
The Median Absolute Deviation (MAD) is often used as a measure of dispersion of the residuals in robust regression. For univariate data, MAD is computed by first finding the median of the data, then ...
3
votes
0answers
224 views
Interpretation of the least absolute deviations linear regression coefficient
In linear regression of $y$ onto $x$, one finds a $\beta_0$ and $\beta_1$ minimizing $\sum \|y - (\beta_1 x + \beta_0)\|^2$. One can show that
$$\beta_1 = \rho(x,y) \frac{\sigma(y)}{\sigma(x)},$$
...
0
votes
1answer
58 views
Only four values per site - how do I detect outliers (with MAD)?
I have four weights. I want to test if there is an outlier within those
four weights.
To me, the number seems quite small to detect outliers.
I thought that the Median Absolute Deviation (MAD) ...
1
vote
1answer
368 views
How to calculate the median absolute deviation for a continuous function?
I know that the median value $(\text{Med})$ for a variable $X$ characterised by the contentious PDF function $f_X(x)$ can be calculated by finding $\text{Med}$ in:
$\int^\text{Med}_{-\infty} dF_X(x) =...
0
votes
1answer
129 views
How does one calculate Fisher-consistency factor for Rousseeuw and Croux's $S_n$ for empirical distribution?
In "Alternatives to the Median Absolute Deviation" (Rousseeuw and Croux, J. Amer. Statistical Assoc, 88(424), 1993, pp.1273–1283), the authors described an estimator of SD better than median absolute ...
5
votes
3answers
2k views
How to normalize if MAD equals zero?
A known way to normalize our feature vectors is:
$$\frac{x_i - \operatorname{median}( X^{(j)})}{\operatorname{MAD}^{(j)}},$$
where $\operatorname{MAD}^{(j)}$ is the median absolute deviation of ...
6
votes
1answer
2k views
Can Median Absolute Deviation (MAD)/SD be used to determine if a distribution is normal or not?
I have recently come across this post on Median Absolute Deviation (MAD). The Wikipedia article here, by the article as an estimator Standard deviation of the distribution is 'k' times MAD, where the ...
0
votes
0answers
34 views
Square of MAD as variance equivalent
I have to estimate the error of a 1D array. The usual way of doing this to take the SD of it and square it.
The issue is the following:
The values in the array represent the time series of the ...
1
vote
1answer
367 views
Why don't dispersions like median deviation and mode deviation exists on the lines of mean deviation?
Or to say in the formula MD=$\frac{1}{N}\sum_{i=1}^{N}|x_i-\overline{x}|$ why can't we have median or mode instead of mean($\overline{x}$) and as a result speak about additional possible measures of ...
8
votes
4answers
1k views
Is there a version of the correlation coefficient that is less-sensitive to outliers?
The correlation coefficient is:
$$ r = \frac{\sum_k \frac{(x_k - \bar{x}) (y_k - \bar{y_k})}{s_x s_y}}{n-1} $$
The sample mean and the sample standard deviation are sensitive to outliers.
As well, ...
1
vote
1answer
164 views
What is the best way to determine which proteins are significantly bound on a testing chip?
I've got a question about the data from a biological experiment.
Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
0
votes
0answers
1k views
What should I use to establish a threshold to detect threshold crossings?
I have data graphed in green here:
I want to detect the peaks but I don't know how to define the threshold values. The blue lines are mean ± standard deviation. Using that, I would miss the three ...
3
votes
0answers
289 views
alternative to IQR [duplicate]
I am currently making a short literature study of robust and efficient estimators. Some very well known are the median absolute deviation (MAD) and the interquartile range (IQR). However they both ...
2
votes
0answers
4k views
When the Median Absolute Deviation (MAD) is zero
Suppose my data look like the following:
(10, 10, 10, 10, 10, 0)
Would it be possible to remove an outlier in this distribution using the median absolute deviation?
Of course, you wouldn't need to ...
0
votes
1answer
67 views
Reducing density of outliers
I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days)
Now the TTB values less than 14 or so are possible. But not with ...
6
votes
1answer
24k views
Determine outliers using IQR or standard deviation?
Similar to ... which doesn't have the answer I'm looking for.
The data set has a normal distribution.
For the project I'm working on, outliers have to be determined over residuals of breeding data....
5
votes
1answer
920 views
Median absolute deviation: impact/bias when forcing a median of zero
I want to estimate the median absolute deviation (MAD) of a signal. The MAD is defined as the median of the absolute difference between the signal and its median.
Now I have a signal that I know for ...
15
votes
2answers
4k views
Median absolute deviation (MAD) and SD of different distributions
For normally distributed data, the standard deviation $\sigma$ and the median absolute deviation $\text{MAD}$ are related by:
$\sigma=\Phi^{-1}(3/4)\cdot \text{MAD}\approx1.4826\cdot\text{MAD},$
...
8
votes
0answers
232 views
Are there non-trivial settings where the MAD statistic has a closed-form density?
The MAD statistic of an iid sample $(x_1,\ldots,x_n)$ is defined as the median of the absolute deviation from the median:
$$
\text{mad}(x_1,\ldots,x_n)=\text{med}\left\{|x_i-\text{med}(x_1,\ldots,x_n)|...
4
votes
2answers
5k views
MAD formula for outlier detection
Does anyone know what is the name of this formula?
$$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$
where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
1
vote
1answer
98 views
Better than Mad, Derive $Q_n$
Could someone be so kind as to walk me through how to do this equation on a dataset?
on pg 5 (marked as 1277) equation 3.3 of this paper.
$$Q_n = c|x_i-x_j|_{(k)},\;\;1\leq i<j\leq n,k\approx{n\...
0
votes
0answers
905 views
Bar plot with median +/- median absolute deviation in scientific publication
I know bar plots usually represents mean +/- sem, but if the data is non-normal, is it ok to bar plot median +/- mad?
I don't like box plots.
Suggestison?
1
vote
0answers
165 views
What's the proper way to do automatic iterative outlier rejection?
I know this is a touchy subject, so I'll proceed with caution...
I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
1
vote
0answers
128 views
Proving the consistancy of the MAD
I am trying to prove that the median absolute deviation from the median (MAD), with k=1.4862, is a consistent estimator of the standard deviation.
1
vote
1answer
204 views
Alternatives to MAD to find a yardstick to assess data
In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is:
$$
Sn = C\, {\...
10
votes
1answer
750 views
What distribution has the maximum entropy for a known mean absolute deviation?
I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum ...
2
votes
1answer
2k views
Understanding MAD Results
I'm using R to calculate the median absolute deviation for a few distributions, but some of the values I'm calculating do not seem realistic at all. I have the following distribution:
...
3
votes
1answer
3k views
MAD in relation to 95% confidence
MAD (Median Absolute Deviation) is:
$\text{MAD} = M_i(|x_i-M_j(x_j)|)$
where $M()$ is the median operator ($M_i(x_i) = \text{median}(x_1,...,x_n)$).
I'd like to scale the MAD in such a way as to ...
14
votes
1answer
1k views
Does a median-unbiased estimator minimize mean absolute deviance?
This is a follow-up but also a different question of my previous one.
I read on Wikipedia that "A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as ...
4
votes
0answers
917 views
How to test stability/variability of statistic over time, using R
I have a long time series data (running several years). I have split the data into several series, each of 1 month duration.
I then perform some custom calculations on each of the 'shortened' time ...
7
votes
2answers
2k views
Median + MAD for skewed data
I am trying to figure out what happens if you apply Hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of Hampel's method over z-...
1
vote
1answer
602 views
Propagation of errors with median absolute deviation from the median?
Is there a theoretically-sound way to perform propagation of errors with robust statistics? I am trying to characterize the errors inherent in a measurement and propagate the uncertainty through ...
2
votes
2answers
3k views
Outlier detection for heavy-tailed data
Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range ...
7
votes
1answer
2k views
Using MAD as a way of defining a threshold for significance testing
If I have a set of terms each term having a particular frequency associated with it (the number of the times the term has appeared in fixed corpus of papers), then is the following method of ...
2
votes
2answers
2k views
MAD equivalent for standard error
As far as I know, one can calculate the relative standard error from the standard deviation of a data sample. I am looking for the Median Absolute Deviation equivalent for standard error.
Does one ...
11
votes
3answers
2k views
Mean$\pm$SD or Median$\pm$MAD to summarise a highly skewed variable?
I'm working on highly skewed data, so I'm using the median instead of the mean to summarise the central tendency. I'd like to have a measure of dispersion
While I often see people reporting mean $\pm$...
4
votes
5answers
3k views
Use of robust spread measures such median average deviation and median filters for time series
I have a time series where I need to detect gross anomalies due to coding errors, not small shifts in the structure of the series. I am interested in the most recent data points, not historical data ...
1
vote
4answers
4k views
How to calculate overall standard deviation from standard deviation of sub-periods?
If I have the mean, s.d., median and count for sample A, and the same for sample B, can I throw away the samples, and calculate exactly mean, median and s.d. for the combination? Well the mean is easy....
6
votes
1answer
3k views
Median of medians as robust mean of means?
The location and scale of a normally distributed data can be estimated by sampling the data then taking the mean of the sample means and standard deviations, respectively. For non-normal (heavy-tailed)...
