Skip to main content
Cross Validated

Timeline for The variance of the weighted median and optimal weights

Current License: CC BY-SA 4.0

35 events
when toggle format what by license comment
Sep 9 at 17:09 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
May 11 at 21:09 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Jan 12 at 9:03 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Sep 10, 2023 at 20:01 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
May 13, 2023 at 0:17 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Jan 7, 2023 at 17:04 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Oct 3, 2022 at 21:00 history bumped CommunityBot This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
May 7, 2022 at 22:47 answer added Wolfgang Brehm timeline score: 0
May 7, 2022 at 21:36 history edited Wolfgang Brehm CC BY-SA 4.0
remove a something irrelevant and wrong
May 5, 2022 at 19:51 history edited Wolfgang Brehm CC BY-SA 4.0
added 2 characters in body
May 5, 2022 at 18:37 history edited Wolfgang Brehm CC BY-SA 4.0
make figure 1 better
May 5, 2022 at 18:24 history edited Wolfgang Brehm CC BY-SA 4.0
highlight the actual question
May 5, 2022 at 14:44 answer added user225256 timeline score: 0
May 4, 2022 at 21:49 history edited Wolfgang Brehm CC BY-SA 4.0
deleted 78 characters in body
May 4, 2022 at 20:31 history edited Wolfgang Brehm CC BY-SA 4.0
Rewriting to highlight the analogy between the weighted mean and the weighted median.
May 4, 2022 at 14:37 comment added Wolfgang Brehm @MattF. I'll add the analogy with the weighted mean then.
May 4, 2022 at 14:32 comment added user225256 If you want a general answer about weighted medians, then it would help to point to a general statement about weighted means and their optimality in the literature, and ask for an analog with medians. But the current post doesn't provide a general statement to use in the comparison, certainly not with a general proof.
May 4, 2022 at 14:27 comment added Wolfgang Brehm @MattF. I don't want a specific answer for my specific problem, because there are even better solutions when we start to model the outliers and distributions involved in greater detail. I want a general answer to how the weighted median behaves and how to derive the weights.
May 4, 2022 at 14:23 comment added user225256 Given your comments, it might be clearer to ask the question as: "We measure an intensity in several ways, each getting a sample $S_i$ with its own error patterns. We take a weighted average of those samples as follows, and reject outliers as follows, and reapply the process without the rejected values as follows, thus estimating the intensity as follows, which is optimal as follows. What would be a similar procedure based on medians rather than means, and with what definitions of outliers and optimality?" But there's a lot of "as follows", and probably not all of it is in the current post.
May 4, 2022 at 14:22 comment added Wolfgang Brehm @whuber in my case I start out with samples coming from the same distribution with an estimate for the minimum standard deviation, but they are multiplied with a scaling constant to get them on the same scale. Whatever distribution you start out with, if you multiply by a constant, the density at the median is divided by the same constant. This is not the only case where you would want a weighted median, but it is the case that started my interest.
May 4, 2022 at 14:16 comment added whuber♦ Fine: but how do you know or estimate the density at the median of each of these distributions? There's some disconnect here, suggesting that you might not have fully explained your situation. What exactly do you know about each $p_i$ and what exactly do you assume about their underlying distributions?
May 4, 2022 at 14:13 comment added Wolfgang Brehm @whuber If the density at the median is a single number weighing the samples differently does not make sense, that is true. But suppose the samples come from different distributions with the same median. Then the density at the median of these distributions that each sample came from is potentially a different number.
May 4, 2022 at 14:05 comment added Wolfgang Brehm @MattF. In my specific case we measure intensities following approximately a normal distribution with variance that can be estimated and some outliers. But the intensity values need to be scaled because each experiment is different, the crystal diffracts more or less, the beam is stronger or weaker. Traditionally we would then use the weighted average to produce a result, and treat the outliers with a $3 \sigma$ rejection criterion, but I would like to compare this with the weighted median because the median of (weighted) means had worked well before.
May 4, 2022 at 13:52 comment added Wolfgang Brehm @MattF. as I said, just like for the weighted average. If all samples had the same distribution the weighted average would make no sense. If the samples have different variance they must have a different distribution. So you have samples with different precision, but there are many outliers too, you would like to use the weighted average, but need something more robust - the weighted median.
May 4, 2022 at 13:52 comment added whuber♦ The density at the median is a single number. I am unable to see any way to use that to develop different weights for different observations. I can imagine that with enough data one might attempt, say, a nonparametric estimation of the mixture density in a neighborhood of the median and then exploit that for weight estimation, but whether that's how you're conceiving of this problem is not apparent. BTW, there's much relevant information at stats.stackexchange.com/questions/45124.
May 4, 2022 at 13:18 comment added user225256 What circumstance leads to all of these distributions having the same median? I can see that happening with symmetric distributions or with lognormal distributions, but in those cases it would be easier to analyze the mean or log-mean.
May 3, 2022 at 20:42 comment added Wolfgang Brehm @whuber just like you only need to know the variance to determine the ideal weights for the weighted average to minimize the variance, it turns out you only need to know the density at the median to determine the weights. I just can't prove it.
May 3, 2022 at 20:33 comment added whuber♦ Then how can you possibly determine the weights with only that information? Are the weights given to you by some oracle that knows the distributions from which the $p_i$ are drawn?
May 3, 2022 at 20:32 history edited Wolfgang Brehm CC BY-SA 4.0
clarify that all distributions need to have the same median
May 3, 2022 at 20:31 comment added Wolfgang Brehm @whuber I should have clarified, with the same median. This median is what I want to find.
May 3, 2022 at 20:28 comment added whuber♦ "Assuming each sample is drawn from a different probability distribution pi," what exactly is your weighted median supposed to be estimating? Unless we know that, it seems we cannot even understand what you might be optimizing.
May 3, 2022 at 20:22 history edited Wolfgang Brehm CC BY-SA 4.0
minor edit: forgot tilde
May 3, 2022 at 12:42 history edited Wolfgang Brehm CC BY-SA 4.0
minor error in ratio of Gaussian to uniform density and formatting
S May 3, 2022 at 12:13 review First questions
May 3, 2022 at 13:17
S May 3, 2022 at 12:13 history asked Wolfgang Brehm CC BY-SA 4.0
toggle format