0
$\begingroup$

Why does manually weighting a regression require the intercept term to be dropped? Consider a model

$$y=b_0 + b_1x + \epsilon, $$

a simple linear regression. In classically weighted regression only the weights are provided, say min(x)/x. In order to supply the weights manually, e.g. to a regression routine that doesn't handle weighting, a vector of weights is created, the square root of the weights is taken, then that gets multiplied to the y's, a vector of 1's, and the x's. The weighted vector of 1's is then included in the model as part of the x-matrix and the model is fit without an intercept. A parameter estimate is returned for x0 that corresponds to the intercept in a weighted regression, even though this term was not constant, and even though the intercept was omitted. Why is this the case?

$\endgroup$
0
0
$\begingroup$

The reason why the intercept is omitted is because you have to manually create the "weighted" intercept analogue. As you have said yourself, this is the vector given by the constant term multiplied by the root of the weights vector. To additionally adjust for the intercept would correspond to a different model.

In other words, a weighted linear regression maximizes the likelihood:

$$ \mathcal{L}_{\alpha, \beta, \sigma}(X, Y, w) = \prod_{i=1}^{n} \phi(Y_i - \alpha - \beta X_i)^{w_i}$$

Where $\phi$ is the normal density.

$$\begin{eqnarray} \phi(Y_i - \alpha - \beta X_i)^{w_i} &\propto& \exp \left(-\frac{1}{2} \left( \frac{Y_i - \alpha -\beta X_i}{\sigma}\right)^2\right) ^{w_i} \\ &=& \exp\left(-\frac{w_i}{2}\left(\frac{Y_i - \alpha -\beta X_i}{\sigma} \right)^2\right) \\ &=& \exp\left(-\frac{1}{2}\left(\frac{\sqrt{w_i}Y_i - \alpha \sqrt{w_i} -\beta \sqrt{w_i}X_i}{\sigma} \right)^2\right) \\ \end{eqnarray} $$

Which is immediately recognizable as the mean model for an OLS regression omitting the intercept and treating the root of the weight vector as the leading covariate, followed by the X_i scaled by the root of the weight vector as the covariable corresponding to the $\beta$ parameter, and Y_i scaled by the root of the weights vector as the response.

$\endgroup$
3
  • $\begingroup$ With respect, I was clearly only asking about linear regression. Excel, for example, doesn't yet handle weighting, even with the data analysis toolkit. There are add-on packages available, though most of these are for pay and include superfluous functionality. "Weights multiplied" is not a trick, it's how regression works with weights, so I don't see how this answers the question. Clearly there is an important difference between the intercept and the coefficients, otherwise they would all be all ones. $\endgroup$ – DCommK May 8 at 1:43
  • $\begingroup$ @DCommK the answer is updated to show the rationale for this approach. $\endgroup$ – AdamO May 12 at 15:43
  • $\begingroup$ Great @AdamO, answer is clear and accepted. Thank you for the explanation! $\endgroup$ – DCommK May 13 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.