5
$\begingroup$

There has been a lot of research about weighted linear regression. Most of the discussions are about weight of samples though, i.e., weight on $y$, rather than weight on $X$,
\begin{align} X &= [X_{0}, X_{1}, ... X_{n}] \\ y &= \beta X \end{align} In most engineering problems, there exist some a-priori knowledge about the factors $X_{i}$. We know that a few factors are 'primary' factors, with strong business logic reasons. For instance, house price ($y$) and median income of zip-code.

I am wondering if there is an approach similar to 'weight of factors' in the regression model, which can introduce above a-priori knowledge.

This should be very useful when we deal with multicollinearity. For a set a correlated factors, we can give higher weight to the factor that we know is important. However, such a-priori knowledge is not really a distribution about $\beta$, but just a weight

Can anyone give any insights here?

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ Bayesian linear regression, for example? $\endgroup$
    – DeltaIV
    Commented Mar 12, 2017 at 22:43
  • $\begingroup$ A doubt: do you have knowledge about the actual values of the $\beta$, or only about the relative importance of the $X_i$? For example, in the house pice case, do you have some prior knowledge that regression coefficient for the median yearly income of zip-code is most likely between 3 and 7, for example? Or your only prior knowledge is that you expect median income to be more important than, say, number of bathrooms, but you couldn't put a band on the plausible values of their regression coefficients? $\endgroup$
    – DeltaIV
    Commented Mar 13, 2017 at 7:39
  • $\begingroup$ concerning your recent edit: as long as you have enough data, it's not like you need to be to extremely confident about your prior distribution, in order to use Bayesian methods! For example, if you would expect $\beta_1$ to be about 3, and you wouldn't be surprised to see a value of 2 or 4, but you would be very surprised to see values of 0 or 6, then $\beta_1\sim\mathcal{N}(3,1)$ could be "good enough". Or, if you only have a very vague idea about the SD of $\beta_1$, you could resort to the usual hierarchical Inverse-Gamma prior on $\sigma$. This is in my opinion the best approach [1/2] $\endgroup$
    – DeltaIV
    Commented Mar 13, 2017 at 14:47
  • $\begingroup$ [2/2] to your problem. But, if you definitely don't want to put priors on $\beta$, I could write an answer about Bayesian Dominance Analysis, where you can put priors on so-called dominance indices, i.e., measures of degree of intrinsic association between predictors and response variables in the presence of other predictors. $\endgroup$
    – DeltaIV
    Commented Mar 13, 2017 at 14:58
  • 1
    $\begingroup$ I think the problem is: how would this weight on the features be used? For example, you could imagine that for each $x_i$, you not only have a $\beta_i$ but also an $\omega_i$: $\omega_i\beta_ix_i$. But that's just the same as scaling $x_i$. $\endgroup$
    – Wayne
    Commented Mar 13, 2017 at 16:41

2 Answers 2

Reset to default
3
$\begingroup$

As @DeltaIV hinted in his or her comment, Bayesian methods are a natural fit for this. Using linear regression in a Bayesian fashion means treating the parameters (viz., the coefficients and the SD of the error term) as random variables. If you think a certain coefficient is more likely to have certain values, you can choose a prior distribution for that random variable that puts more weight on the values in question.

Improve this answer
$\endgroup$
0
$\begingroup$

I'm not aware of any method which takes hints like "Variable X1 is more important than Variable X3".

As the comments to the OP reflect, this isn't straightforwardly a Bayesian regression issue, since you're interested in weighting the inclusion of variables, not (directly) in weighting possible values of the coefficients.

Three thoughts on what could be done:

  1. Use Bayesian Regression and give your preferred variables the usual priors, while giving non-preferred variables sparsity-inducing priors such as the Horseshoe. I don't know if this would actually work, but the idea would be to give the non-preferred variables more opportunity to have their coefficients be driven to zero.

  2. Modify a LARS-style regression to weight the angles inversely-proportionally to whether they are favored variables or not.

  3. Modify a stepwise regression (I don't like stepwise, but it would be easier to modify) such that it prefers to add the preferred variables first. Or perhaps starts with the preferred variables in and then adds or subtracts the other variables in the steps. (See this other posting.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ "this isn't straightforwardly a Bayesian regression issue, since you're interested in weighting the inclusion of variables, not (directly) in weighting possible values of the coefficients" — But excluding a variable is equivalent to giving it a coefficient of 0. So you could give a nonzero prior weight to {0} to represent a prior belief that the variable doesn't belong. $\endgroup$
    – Kodiologist
    Commented Mar 15, 2017 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.