0
$\begingroup$

For a simple linear regression model without intercept, that is $$y_i=ax_i+\varepsilon_i$$ where $\varepsilon_i\sim_{iid} N(0, \tau^2), i=1,2,\dots, n$ and $x_i$ is a fixed covariate. Assume that $a|\tau \sim N(\mu, \tau^2)$. and $\tau$ is fixed.

By the least squares estimator of $a$, we know that the minimum of $$\sum_{i=1}^n(y_i-\hat{a}x_i)^2$$ obtained at the estimator of $a$ $$\hat{a}=\frac{\sum_{i}x_iy_i}{\sum_i x_i^2}$$

But I do not know how to use this estimator to estimate the $\mu$?

$\endgroup$
2
  • $\begingroup$ What does "$\gamma$" refer to?? $\endgroup$
    – whuber
    Commented Oct 21, 2020 at 13:12
  • $\begingroup$ @whuber Sorry, this is $\tau$. $\endgroup$
    – user261225
    Commented Oct 22, 2020 at 0:11

1 Answer 1

1
$\begingroup$

It looks to me like $\mathcal{N}(\mu,\tau^2)$ is the prior distribution of $a$. Thus, you don't estimate $\mu$, it is a parameter that describes your initial belief of $a$ -- often this is set to $0$ to encourage low weights unless you have reason to believe another mean is better.

So after you see more data, you compute the posterior mean of $a$ using bayes rule. Then you can either set $a$ as the argmax of the posterior and perform linear regression (MAP estimate), or you can integrate out $\mu$ to derive a predictive distribution of $y$, again using bayes rule.

$\endgroup$
2
  • $\begingroup$ I don't understand the reason for setting $\mu=0,$ because it looks like all numbers are equally valid for the prior. After all, for any real number $\lambda$ this model is identical to the model $$z_i = (a-\lambda)x_i + \varepsilon_i$$ where $z_i = y_i - \lambda x_i.$ If your reasoning is correct, than in the re-expressed model we would stipulate $a-\lambda=0,$ which is the same as $a=\lambda.$ How do you resolve this apparent contradiction? $\endgroup$
    – whuber
    Commented Oct 22, 2020 at 12:50
  • 1
    $\begingroup$ I don't think this is a contradiction, if I understand you correctly. Generally you prefer to keep your parameters small in norm to avoid overfitting, which is why I would set mu to 0 unless I had reason to believe otherwise. This is actually equivalent to performing l2 weight decay. $\endgroup$
    – harwiltz
    Commented Oct 22, 2020 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.