Questions tagged [mad]

Median Absolute Deviation (MAD) is a measure of variability in a sample of data, and is often used as an alternative to measures like standard deviation since it is more resistant to outliers. Use the [mae] tag if you are asking about the point forecast accuracy measure called MAD or MAE.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Comparing Star Ratings

Data Illustration NOTE: I do not have a stats background. I'm pretty sure I'll use some terminology incorrectly. I have come across several approaches via Python/SQL that I would like to confirm as ...
1 vote
0 answers
66 views

MAD & Median of weighted GMM

What is the median and median-absolute-deviation of a weighted GMM in terms of component mean and variance? For example, three normal distributions $A$, $B$, $C$ with means $\mu_a,\mu_b,\mu_c$, ...
0 votes
0 answers
34 views

Constructing a User Profile for Music Taste

My goal is to construct User profiles based on positive (and maybe also negative) interactions with songs. A User has the option to like a song. This would give me a list of likes for each user. With ...
1 vote
1 answer
113 views

Method for outlier detection in noisy seasonal time series data?

I have around 1000 times series of around 1000 samples, where each sample is 5 minutes a part. An example of a time series after performing seasonal decomposition is As we can see the data is very ...
  • 151
0 votes
1 answer
43 views

Mean and Mean Absolute Deviation inferential statistics

Consider the following data of runners in a race. The first data is from a certain year the second one is from the year after. $Last\space year \space|\space Mean: 525s \space|\space MAD: 25s$ $This \...
  • 105
1 vote
1 answer
3k views

Calculating robust z scores with median and MAD

Could someone explain the scaling factors involved in calculating robust z scores using median and MAD please? As I understand it, conventional Z scores calculated using the mean and SD are sensitive ...
  • 13
3 votes
1 answer
2k views

Median absolute deviation only can be used for anomaly detection for time series without a trend?

I think MAD only can be used to detect anomalies for time series without a trend because it relies only a stable median to detect anomalies. It should be OK for time series with seasonality. Just seek ...
  • 867
2 votes
2 answers
1k views

Replacing outlier removal from IQR to MAD

A common outlier removal formula is Q3 + IQR * 1.5 and Q1 - IQR * 1.5 Outliers can also be removed using Mean Absolute Deviation and Median Absolute Deviation. Is ...
  • 532
1 vote
0 answers
32 views

Check which parameter causes less dispersion in a sample set - When to use CV (Coefficient of Variation) or CD (Coefficient of Dispersion) etc

The overall goal is to extract/engineer features from approximately 100 segments (various lengths but always more than 80 data points) that are as similar as possible to each other and have a very low ...
1 vote
0 answers
177 views

rolling removing outliers: include or not include

In the paper "Realized kernels in practice: trades and quotes" by O. E.Bandorff-Nielsen etc. cf. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1368-423X.2008.00275.x in the section dedicated to ...
  • 376
1 vote
0 answers
43 views

Mean Absolute Deviation and data preprocessing

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Next, based on Mean Absolute Deviation (MAD) we aim to decide if the last point $x_{n+1}$ is outlier or not. First, let us compute the MAD: ...
  • 376
0 votes
0 answers
469 views

Convert median absolute deviation (MAD) to SD for log-normal distribution

How do I convert the $\text{MAD}$ (median absolute deviation from the median) of data that is drawn from a log-normal distribution to the standard deviation of a log-normal distribution? To clarify, ...
  • 41
1 vote
3 answers
1k views

What's the best method to measure relative variability for non normal data?

I have a set of data that states the volume required each month for the next 12 for 750 raw materials. I would like to determine the variability in the demand for each material, and categorise the ...
  • 11
1 vote
1 answer
91 views

Inference using robust statistics

I'm simulating the overall usage of a cluster using historic deployment data. Due to the nature of the simulation, there are some heavy points (i.e. very low overall usage). As a result, the variance ...
0 votes
1 answer
317 views

Welch t-test using the median and MAD

Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers. I'm using this to test ...
1 vote
0 answers
1k views

Median absolute deviation outlier detection for new data points

I have been reading this article for outlier detection and the method proposed works really well when applied over my data set. However, it would be great if i could run outlier check for new data ...
  • 343
5 votes
1 answer
4k views

How to estimate the scale factor for MAD for a non-normal distribution?

I understand that the scale factor for normally distributed data is 1.4826 to convert it to a pseudo standard deviation like quantity which could be used with the median for determining confidence ...
2 votes
1 answer
853 views

MAD & Standard Deviation

Am I correct in saying that you can only use the formula for approximating the standard deviation from MAD, i.e $$\text{SD } = K \times \text{ MAD }$$ if you know the actual probability ...
3 votes
1 answer
473 views

How to prove 'For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution'

For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution I came across this statement in the wikipedia page for MAD(Median Absolute Deviation) but ...
's user avatar
2 votes
0 answers
140 views

Robust Regression

The Median Absolute Deviation (MAD) is often used as a measure of dispersion of the residuals in robust regression. For univariate data, MAD is computed by first finding the median of the data, then ...
  • 21
4 votes
0 answers
504 views

Interpretation of the least absolute deviations linear regression coefficient

In linear regression of $y$ onto $x$, one finds a $\beta_0$ and $\beta_1$ minimizing $\sum \|y - (\beta_1 x + \beta_0)\|^2$. One can show that $$\beta_1 = \rho(x,y) \frac{\sigma(y)}{\sigma(x)},$$ ...
0 votes
1 answer
138 views

Only four values per site - how do I detect outliers (with MAD)?

I have four weights. I want to test if there is an outlier within those four weights. To me, the number seems quite small to detect outliers. I thought that the Median Absolute Deviation (MAD) ...
4 votes
1 answer
614 views

How to calculate the median absolute deviation for a continuous function?

I know that the median value $(\text{Med})$ for a variable $X$ characterised by the contentious PDF function $f_X(x)$ can be calculated by finding $\text{Med}$ in: $\int^\text{Med}_{-\infty} dF_X(x) =...
1 vote
1 answer
210 views

How does one calculate Fisher-consistency factor for Rousseeuw and Croux's $S_n$ for empirical distribution?

In "Alternatives to the Median Absolute Deviation" (Rousseeuw and Croux, J. Amer. Statistical Assoc, 88(424), 1993, pp.1273–1283), the authors described an estimator of SD better than median absolute ...
  • 549
5 votes
3 answers
3k views

How to normalize if MAD equals zero?

A known way to normalize our feature vectors is: $$\frac{x_i - \operatorname{median}( X^{(j)})}{\operatorname{MAD}^{(j)}},$$ where $\operatorname{MAD}^{(j)}$ is the median absolute deviation of ...
7 votes
1 answer
3k views

Can Median Absolute Deviation (MAD)/SD be used to determine if a distribution is normal or not?

I have recently come across this post on Median Absolute Deviation (MAD). The Wikipedia article here, by the article as an estimator Standard deviation of the distribution is 'k' times MAD, where the ...
  • 198
0 votes
0 answers
60 views

Square of MAD as variance equivalent

I have to estimate the error of a 1D array. The usual way of doing this to take the SD of it and square it. The issue is the following: The values in the array represent the time series of the ...
1 vote
1 answer
443 views

Why don't dispersions like median deviation and mode deviation exists on the lines of mean deviation?

Or to say in the formula MD=$\frac{1}{N}\sum_{i=1}^{N}|x_i-\overline{x}|$ why can't we have median or mode instead of mean($\overline{x}$) and as a result speak about additional possible measures of ...
  • 351
12 votes
4 answers
3k views

Is there a version of the correlation coefficient that is less-sensitive to outliers?

The correlation coefficient is: $$ r = \frac{\sum_k \frac{(x_k - \bar{x}) (y_k - \bar{y_k})}{s_x s_y}}{n-1} $$ The sample mean and the sample standard deviation are sensitive to outliers. As well, ...
1 vote
1 answer
210 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
  • 13
0 votes
0 answers
2k views

What should I use to establish a threshold to detect threshold crossings?

I have data graphed in green here: I want to detect the peaks but I don't know how to define the threshold values. The blue lines are mean ± standard deviation. Using that, I would miss the three ...
  • 101
3 votes
0 answers
597 views

alternative to IQR [duplicate]

I am currently making a short literature study of robust and efficient estimators. Some very well known are the median absolute deviation (MAD) and the interquartile range (IQR). However they both ...
4 votes
0 answers
7k views

When the Median Absolute Deviation (MAD) is zero

Suppose my data look like the following: (10, 10, 10, 10, 10, 0) Would it be possible to remove an outlier in this distribution using the median absolute deviation? Of course, you wouldn't need to ...
  • 49
0 votes
1 answer
71 views

Reducing density of outliers

I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days) Now the TTB values less than 14 or so are possible. But not with ...
10 votes
1 answer
34k views

Determine outliers using IQR or standard deviation?

Similar to ... which doesn't have the answer I'm looking for. The data set has a normal distribution. For the project I'm working on, outliers have to be determined over residuals of breeding data....
  • 223
5 votes
1 answer
1k views

Median absolute deviation: impact/bias when forcing a median of zero

I want to estimate the median absolute deviation (MAD) of a signal. The MAD is defined as the median of the absolute difference between the signal and its median. Now I have a signal that I know for ...
  • 147
16 votes
2 answers
6k views

Median absolute deviation (MAD) and SD of different distributions

For normally distributed data, the standard deviation $\sigma$ and the median absolute deviation $\text{MAD}$ are related by: $\sigma=\Phi^{-1}(3/4)\cdot \text{MAD}\approx1.4826\cdot\text{MAD},$ ...
  • 161
14 votes
1 answer
374 views

Are there non-trivial settings where the MAD statistic has a closed-form density?

The MAD statistic of an iid sample $(x_1,\ldots,x_n)$ is defined as the median of the absolute deviation from the median: $$ \text{mad}(x_1,\ldots,x_n)=\text{med}\left\{|x_i-\text{med}(x_1,\ldots,x_n)|...
  • 94.5k
15 votes
2 answers
12k views

MAD formula for outlier detection

Does anyone know what is the name of this formula? $$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$ where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
  • 153
1 vote
1 answer
111 views

Better than Mad, Derive $Q_n$

Could someone be so kind as to walk me through how to do this equation on a dataset? on pg 5 (marked as 1277) equation 3.3 of this paper. $$Q_n = c|x_i-x_j|_{(k)},\;\;1\leq i<j\leq n,k\approx{n\...
0 votes
0 answers
1k views

Bar plot with median +/- median absolute deviation in scientific publication

I know bar plots usually represents mean +/- sem, but if the data is non-normal, is it ok to bar plot median +/- mad? I don't like box plots. Suggestison?
1 vote
0 answers
212 views

What's the proper way to do automatic iterative outlier rejection?

I know this is a touchy subject, so I'll proceed with caution... I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
1 vote
0 answers
190 views

Proving the consistancy of the MAD

I am trying to prove that the median absolute deviation from the median (MAD), with k=1.4862, is a consistent estimator of the standard deviation.
1 vote
1 answer
229 views

Alternatives to MAD to find a yardstick to assess data

In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is: $$ Sn = C\, {\...
12 votes
1 answer
1k views

What distribution has the maximum entropy for a known mean absolute deviation?

I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum ...
2 votes
1 answer
2k views

Understanding MAD Results

I'm using R to calculate the median absolute deviation for a few distributions, but some of the values I'm calculating do not seem realistic at all. I have the following distribution: ...
  • 244
5 votes
2 answers
5k views

MAD in relation to 95% confidence

MAD (Median Absolute Deviation) is: $\text{MAD} = M_i(|x_i-M_j(x_j)|)$ where $M()$ is the median operator ($M_i(x_i) = \text{median}(x_1,...,x_n)$). I'd like to scale the MAD in such a way as to ...
  • 244
14 votes
1 answer
2k views

Does a median-unbiased estimator minimize mean absolute deviance?

This is a follow-up but also a different question of my previous one. I read on Wikipedia that "A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as ...
  • 393
4 votes
0 answers
1k views

How to test stability/variability of statistic over time, using R

I have a long time series data (running several years). I have split the data into several series, each of 1 month duration. I then perform some custom calculations on each of the 'shortened' time ...
8 votes
2 answers
3k views

Median + MAD for skewed data

I am trying to figure out what happens if you apply Hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of Hampel's method over z-...