Questions tagged [mad]

Median Absolute Deviation (MAD) is a measure of variability in a sample of data, and is often used as an alternative to measures like standard deviation since it is more resistant to outliers. Use the [mae] tag if you are asking about the point forecast accuracy measure called MAD or MAE.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Comparing Star Ratings

Data Illustration NOTE: I do not have a stats background. I'm pretty sure I'll use some terminology incorrectly. I have come across several approaches via Python/SQL that I would like to confirm as ...
user avatar
1 vote
0 answers
57 views

MAD & Median of weighted GMM

What is the median and median-absolute-deviation of a weighted GMM in terms of component mean and variance? For example, three normal distributions $A$, $B$, $C$ with means $\mu_a,\mu_b,\mu_c$, ...
user avatar
0 votes
0 answers
23 views

Constructing a User Profile for Music Taste

My goal is to construct User profiles based on positive (and maybe also negative) interactions with songs. A User has the option to like a song. This would give me a list of likes for each user. With ...
user avatar
1 vote
1 answer
58 views

Method for outlier detection in noisy seasonal time series data?

I have around 1000 times series of around 1000 samples, where each sample is 5 minutes a part. An example of a time series after performing seasonal decomposition is As we can see the data is very ...
user avatar
  • 151
0 votes
1 answer
38 views

Mean and Mean Absolute Deviation inferential statistics

Consider the following data of runners in a race. The first data is from a certain year the second one is from the year after. $Last\space year \space|\space Mean: 525s \space|\space MAD: 25s$ $This \...
user avatar
  • 105
0 votes
0 answers
30 views

Maximum Absolute Deviation Test - Change of rank?

For the point process finpines (standard data set of the spatstat library), I would like to verify whether it satisfies the ...
user avatar
  • 1
1 vote
1 answer
2k views

Calculating robust z scores with median and MAD

Could someone explain the scaling factors involved in calculating robust z scores using median and MAD please? As I understand it, conventional Z scores calculated using the mean and SD are sensitive ...
user avatar
  • 13
3 votes
1 answer
1k views

Median absolute deviation only can be used for anomaly detection for time series without a trend?

I think MAD only can be used to detect anomalies for time series without a trend because it relies only a stable median to detect anomalies. It should be OK for time series with seasonality. Just seek ...
user avatar
  • 643
2 votes
2 answers
904 views

Replacing outlier removal from IQR to MAD

A common outlier removal formula is Q3 + IQR * 1.5 and Q1 - IQR * 1.5 Outliers can also be removed using Mean Absolute Deviation and Median Absolute Deviation. Is ...
user avatar
  • 532
1 vote
0 answers
29 views

Check which parameter causes less dispersion in a sample set - When to use CV (Coefficient of Variation) or CD (Coefficient of Dispersion) etc

The overall goal is to extract/engineer features from approximately 100 segments (various lengths but always more than 80 data points) that are as similar as possible to each other and have a very low ...
user avatar
1 vote
0 answers
157 views

rolling removing outliers: include or not include

In the paper "Realized kernels in practice: trades and quotes" by O. E.Bandorff-Nielsen etc. cf. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1368-423X.2008.00275.x in the section dedicated to ...
user avatar
  • 406
1 vote
0 answers
36 views

Mean Absolute Deviation and data preprocessing

Assume we have data points $x_{1}, \dots, x_{n}, x_{n+1}$. Next, based on Mean Absolute Deviation (MAD) we aim to decide if the last point $x_{n+1}$ is outlier or not. First, let us compute the MAD: ...
user avatar
  • 406
0 votes
0 answers
417 views

Convert median absolute deviation (MAD) to SD for log-normal distribution

How do I convert the $\text{MAD}$ (median absolute deviation from the median) of data that is drawn from a log-normal distribution to the standard deviation of a log-normal distribution? To clarify, ...
user avatar
  • 41
1 vote
3 answers
973 views

What's the best method to measure relative variability for non normal data?

I have a set of data that states the volume required each month for the next 12 for 750 raw materials. I would like to determine the variability in the demand for each material, and categorise the ...
user avatar
  • 11
1 vote
1 answer
86 views

Inference using robust statistics

I'm simulating the overall usage of a cluster using historic deployment data. Due to the nature of the simulation, there are some heavy points (i.e. very low overall usage). As a result, the variance ...
user avatar
0 votes
1 answer
296 views

Welch t-test using the median and MAD

Can the Welch t-test be used with the median and MAD instead of mean and variance? I think outliers are causing problems and the median places less weight on extreme outliers. I'm using this to test ...
user avatar
1 vote
0 answers
1k views

Median absolute deviation outlier detection for new data points

I have been reading this article for outlier detection and the method proposed works really well when applied over my data set. However, it would be great if i could run outlier check for new data ...
user avatar
  • 313
2 votes
1 answer
778 views

MAD & Standard Deviation

Am I correct in saying that you can only use the formula for approximating the standard deviation from MAD, i.e $$\text{SD } = K \times \text{ MAD }$$ if you know the actual probability ...
user avatar
3 votes
1 answer
429 views

How to prove 'For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution'

For a symmetric distribution with zero mean, the population MAD is the 75th percentile of the distribution I came across this statement in the wikipedia page for MAD(Median Absolute Deviation) but ...
user avatar
2 votes
0 answers
120 views

Robust Regression

The Median Absolute Deviation (MAD) is often used as a measure of dispersion of the residuals in robust regression. For univariate data, MAD is computed by first finding the median of the data, then ...
user avatar
  • 21
4 votes
0 answers
484 views

Interpretation of the least absolute deviations linear regression coefficient

In linear regression of $y$ onto $x$, one finds a $\beta_0$ and $\beta_1$ minimizing $\sum \|y - (\beta_1 x + \beta_0)\|^2$. One can show that $$\beta_1 = \rho(x,y) \frac{\sigma(y)}{\sigma(x)},$$ ...
user avatar
0 votes
1 answer
131 views

Only four values per site - how do I detect outliers (with MAD)?

I have four weights. I want to test if there is an outlier within those four weights. To me, the number seems quite small to detect outliers. I thought that the Median Absolute Deviation (MAD) ...
user avatar
3 votes
1 answer
559 views

How to calculate the median absolute deviation for a continuous function?

I know that the median value $(\text{Med})$ for a variable $X$ characterised by the contentious PDF function $f_X(x)$ can be calculated by finding $\text{Med}$ in: $\int^\text{Med}_{-\infty} dF_X(x) =...
user avatar
1 vote
1 answer
195 views

How does one calculate Fisher-consistency factor for Rousseeuw and Croux's $S_n$ for empirical distribution?

In "Alternatives to the Median Absolute Deviation" (Rousseeuw and Croux, J. Amer. Statistical Assoc, 88(424), 1993, pp.1273–1283), the authors described an estimator of SD better than median absolute ...
user avatar
  • 519
5 votes
3 answers
3k views

How to normalize if MAD equals zero?

A known way to normalize our feature vectors is: $$\frac{x_i - \operatorname{median}( X^{(j)})}{\operatorname{MAD}^{(j)}},$$ where $\operatorname{MAD}^{(j)}$ is the median absolute deviation of ...
user avatar
7 votes
1 answer
3k views

Can Median Absolute Deviation (MAD)/SD be used to determine if a distribution is normal or not?

I have recently come across this post on Median Absolute Deviation (MAD). The Wikipedia article here, by the article as an estimator Standard deviation of the distribution is 'k' times MAD, where the ...
user avatar
  • 198
0 votes
0 answers
56 views

Square of MAD as variance equivalent

I have to estimate the error of a 1D array. The usual way of doing this to take the SD of it and square it. The issue is the following: The values in the array represent the time series of the ...
user avatar
1 vote
1 answer
432 views

Why don't dispersions like median deviation and mode deviation exists on the lines of mean deviation?

Or to say in the formula MD=$\frac{1}{N}\sum_{i=1}^{N}|x_i-\overline{x}|$ why can't we have median or mode instead of mean($\overline{x}$) and as a result speak about additional possible measures of ...
user avatar
  • 351
12 votes
4 answers
3k views

Is there a version of the correlation coefficient that is less-sensitive to outliers?

The correlation coefficient is: $$ r = \frac{\sum_k \frac{(x_k - \bar{x}) (y_k - \bar{y_k})}{s_x s_y}}{n-1} $$ The sample mean and the sample standard deviation are sensitive to outliers. As well, ...
user avatar
1 vote
1 answer
203 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
user avatar
  • 13
0 votes
0 answers
2k views

What should I use to establish a threshold to detect threshold crossings?

I have data graphed in green here: I want to detect the peaks but I don't know how to define the threshold values. The blue lines are mean ± standard deviation. Using that, I would miss the three ...
user avatar
  • 101
3 votes
0 answers
596 views

alternative to IQR [duplicate]

I am currently making a short literature study of robust and efficient estimators. Some very well known are the median absolute deviation (MAD) and the interquartile range (IQR). However they both ...
user avatar
4 votes
0 answers
7k views

When the Median Absolute Deviation (MAD) is zero

Suppose my data look like the following: (10, 10, 10, 10, 10, 0) Would it be possible to remove an outlier in this distribution using the median absolute deviation? Of course, you wouldn't need to ...
user avatar
  • 49
0 votes
1 answer
71 views

Reducing density of outliers

I have a dataset with right skewed data. It represent frequency of candidates vs their TTB (which is essentially the number of days) Now the TTB values less than 14 or so are possible. But not with ...
user avatar
10 votes
1 answer
33k views

Determine outliers using IQR or standard deviation?

Similar to ... which doesn't have the answer I'm looking for. The data set has a normal distribution. For the project I'm working on, outliers have to be determined over residuals of breeding data....
user avatar
  • 223
5 votes
1 answer
1k views

Median absolute deviation: impact/bias when forcing a median of zero

I want to estimate the median absolute deviation (MAD) of a signal. The MAD is defined as the median of the absolute difference between the signal and its median. Now I have a signal that I know for ...
user avatar
  • 147
16 votes
2 answers
6k views

Median absolute deviation (MAD) and SD of different distributions

For normally distributed data, the standard deviation $\sigma$ and the median absolute deviation $\text{MAD}$ are related by: $\sigma=\Phi^{-1}(3/4)\cdot \text{MAD}\approx1.4826\cdot\text{MAD},$ ...
user avatar
  • 161
13 votes
1 answer
357 views

Are there non-trivial settings where the MAD statistic has a closed-form density?

The MAD statistic of an iid sample $(x_1,\ldots,x_n)$ is defined as the median of the absolute deviation from the median: $$ \text{mad}(x_1,\ldots,x_n)=\text{med}\left\{|x_i-\text{med}(x_1,\ldots,x_n)|...
user avatar
  • 91.8k
14 votes
2 answers
11k views

MAD formula for outlier detection

Does anyone know what is the name of this formula? $$M_i = \displaystyle\frac{0.6745(x_i - \hat{x})}{\mathrm{MAD}}$$ where $\textrm{MAD}$ is the median absolute deviation and $\hat{x}$ is the median ...
user avatar
  • 143
1 vote
1 answer
107 views

Better than Mad, Derive $Q_n$

Could someone be so kind as to walk me through how to do this equation on a dataset? on pg 5 (marked as 1277) equation 3.3 of this paper. $$Q_n = c|x_i-x_j|_{(k)},\;\;1\leq i<j\leq n,k\approx{n\...
user avatar
0 votes
0 answers
1k views

Bar plot with median +/- median absolute deviation in scientific publication

I know bar plots usually represents mean +/- sem, but if the data is non-normal, is it ok to bar plot median +/- mad? I don't like box plots. Suggestison?
user avatar
1 vote
0 answers
208 views

What's the proper way to do automatic iterative outlier rejection?

I know this is a touchy subject, so I'll proceed with caution... I am building regression models for some hundreds of thousands of data files containing only two columns. The sensor collecting these ...
user avatar
1 vote
0 answers
176 views

Proving the consistancy of the MAD

I am trying to prove that the median absolute deviation from the median (MAD), with k=1.4862, is a consistent estimator of the standard deviation.
user avatar
1 vote
1 answer
226 views

Alternatives to MAD to find a yardstick to assess data

In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is: $$ Sn = C\, {\...
user avatar
12 votes
1 answer
1k views

What distribution has the maximum entropy for a known mean absolute deviation?

I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum ...
user avatar
2 votes
1 answer
2k views

Understanding MAD Results

I'm using R to calculate the median absolute deviation for a few distributions, but some of the values I'm calculating do not seem realistic at all. I have the following distribution: ...
user avatar
  • 244
5 votes
2 answers
5k views

MAD in relation to 95% confidence

MAD (Median Absolute Deviation) is: $\text{MAD} = M_i(|x_i-M_j(x_j)|)$ where $M()$ is the median operator ($M_i(x_i) = \text{median}(x_1,...,x_n)$). I'd like to scale the MAD in such a way as to ...
user avatar
  • 244
14 votes
1 answer
1k views

Does a median-unbiased estimator minimize mean absolute deviance?

This is a follow-up but also a different question of my previous one. I read on Wikipedia that "A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as ...
user avatar
  • 393
4 votes
0 answers
1k views

How to test stability/variability of statistic over time, using R

I have a long time series data (running several years). I have split the data into several series, each of 1 month duration. I then perform some custom calculations on each of the 'shortened' time ...
user avatar
8 votes
2 answers
3k views

Median + MAD for skewed data

I am trying to figure out what happens if you apply Hampel's outlier detection technique based on the median and the MAD to data that is skewed. Apparently, the advantage of Hampel's method over z-...
user avatar