Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite
1

I'm performing a weighted linear least squares fit, where the weights correspond to the number of counts of a specific observation. Due to the nature of the data, it is possible that a small handful of observations get weighted much more than the rest, so that the regression is dominated by, say, two data points only, which obviously leads to bad results.

I've run a few tests and I noticed that by e.g. capping the weights, I can improve the results, though this feels rather hack-ish to me. Are there better ways to avoid a small number of data points to out-weigh the rest of the data, or, if capping the weights is a good approach, is there non-ad hoc way to determine the optimal value of the cap?

EDIT

Here's what some sample data and fits look like with (a) no weights, and (b) weights. The data were generated from fairly realistic simulations, so I know the ground truth (red line).

enter image description here enter image description here

My problem is that the weight of some data points (#2 at x=10 for example) can be sufficiently large to dominate the fit. However, I also don't want the very-low-count data to weigh in too much, otherwise I get a really crappy fit as well.

share|improve this question
Have you tried applying a function to the weight? Reasonable choices would be a log or sigmoid. – rm999 Jan 11 '12 at 17:08
3  
If you don't like the weights, why are you using them? Capping the weights is esentially using a different weighting function, and is also suspect in the sense that you are in effect giving more weight to the infrequent events (typically, outliers) than to the most frequent events. "Improving" the results by modifying the weights also makes me wonder whether you have a predetermined result to arrive at in mind, and are trying to come up with a weighting function that will achieve the result that you want to get. Else, how do you know that changing the weights improves the results? – Dilip Sarwate Jan 11 '12 at 17:42
4  
Is it really the case that a small handful of observations is getting weighted more heavily if the weight is the number of counts of the observation? From your description, it seems to me that each physical observation is really $n_i$ actual observations, in which case each actual observation is getting a weight of one (using your weighting scheme.) – jbowman Jan 11 '12 at 18:19
1  
Is there something different about the frequently occurring observations that you could account for in the model? i.e. a reason why you may expect them to be "different" to the rest? If not, then you cannot reasonably claim that they are "skewing" the results. – probabilityislogic Jan 11 '12 at 20:40
1  
@DilipSarwate: Since I'm testing this on simulated data, I know exactly which result I should expect, and I want to find out how I can get a trustworthy fit to data I know before I start working with the real data coming off the sensor. Also, you're right in that I'm using the weights to penalize the low-weight data, so making them more important is indeed an issue. – Jonas Jan 12 '12 at 17:13
show 6 more comments

1 Answer

up vote 1 down vote accepted

You can try applying a function to the weight. Reasonable choices would be a log or sigmoid.

share|improve this answer
Thanks for the suggestion. Log won't work in my case, because it gives too much weight for the low-count data, but sigmoid seems to be exactly what I was looking for. Is there any good rule of thumb for choosing the transition point? – Jonas Jan 16 '12 at 13:57
This sure sounds ad hoc to me. – whuber♦ Jan 16 '12 at 15:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.