1
$\begingroup$

Consider the basic linear regression model $y = A \theta$ with $y\in R^n$ and $A \in R^{n\times k}$ measurements and $\theta \in R^k$ parameters to be estimated. In my case, $\theta$ are physically meaningful parameters that I would like to estimate for further interpretation.

A situation that I encounter frequently is that $A$ and $y$ contain time series of different measurement modalities that are rather repetitive over time, with some rare "interesting" anomalies occurring from time to time. Basic linear regression will weight all measurements equally, hence leading to a model that explains the standard measurements (occurring frequently) well, but may fail during unusual events.

Consequently, I thought it rather natural to perform a weighted regression, where measurements are weighted by the inverse of their likelihood of occurrence, hence weighting unusual measurements stronger than with basic (unweighted) regression and hopefully leading to a model that is more generally applicable. More precisely, I would like to assign a weight $w_i = f(p(y_i, a_i|y, A))$ to each measurement, where $p(y_i,a_i|y,A)$ denotes the likelihood of a new measurement taking values $y_i$ and $a_i$, having observed all the data in $y$ and $A$, and $f(x)$ is a positive, monotonously decreasing function for $x>0$, e.g. $f(x)=1/x$. However, I seem to be unable to find any sources doing something like this, which made me wonder: Is this ...

  1. a standard thing that people do and I'm simply unable to recognize it or find the right terms for searching for it,
  2. completely unreasonable for some reason, or
  3. reasonable but nobody has done it (seems unlikely)?

Disclaimer: I'm a mathematician / engineer / computer scientist and mostly self-taught statistician, so please bear with me if this is completely obvious...

$\endgroup$
  • 1
    $\begingroup$ Could you explain how the "likelihood of occurrence" is estimated or computed? And how do you account for the correlations among the components of $y$ in your fitting procedure? What is the purpose of this regression? $\endgroup$ – whuber Jan 31 '18 at 22:32
  • $\begingroup$ You might want to read up on "leverage" in regression. Points in $A$ that are far from its mean will in fact potentially have quite an impact on the parameter estimates by standard estimation methods, so the broad approach you're suggesting is in fact more commonly used to give less weight to "unusual" values of the explanatory variables. $\endgroup$ – Peter Ellis Jan 31 '18 at 22:40
  • $\begingroup$ @whuber Currently, the likelihood isn't estimated at all, as I'm still trying to understand whether the general approach is useful or not. But I guess one could do something like a multivariate kernel density estimation? Also, I don't (yet) account for the correlations among the components of $y$. Thanks for the pointer; I did some quick reading and guess that I should be doing something like Feasible GLS? The purpose of the regression is to estimate physically meaningful parameters from a simple linear model for further interpretation in other contexts. $\endgroup$ – jhin Jan 31 '18 at 23:16
  • $\begingroup$ @PeterEllis Thanks, that is indeed interesting and I will take a deeper look at it to analyze the effect my measurements have on the outcome. However, the underlying issue remains that the overall influence of measurements in a certain range is determined by the amount of data in this range. This effect is what I would like to get rid of, in order to obtain a model that works equally well over the full data range. $\endgroup$ – jhin Jan 31 '18 at 23:24
  • $\begingroup$ I'm still unable to determine what you might mean by "likelihood." Because that's crucial to understanding your question, editing it to clarify this would be most helpful. $\endgroup$ – whuber Jan 31 '18 at 23:36
1
$\begingroup$

Something similar does occur, but I'm really not sure whether it would apply to your scenario. I'm not aware of weighting by an attempted probabilistic measure (what you call "likelihood of occurrence").

The similar procedures I'm referring to are to weight by relative frequency or by inverse-variance. This article gives examples of doing so for several different linear models.

$\endgroup$
  • 1
    $\begingroup$ Thanks, I had already stumbled upon similar sources myself. The problem, however, is that in these settings the variance of the individual measurements is always assumed to be somehow known a priori, and not something learned from the data themselves. $\endgroup$ – jhin Feb 2 '18 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.