8
$\begingroup$

SE community, I hope to get some insights into the following problem. Given a simple linear regression model $$Y=X\beta+\epsilon\text{ , where } Y\in\mathbb{R}^T,X\in\mathbb{R}^{T \times N}.$$ Under a Gaussian likelihood function with homoscedastic error terms the conditional distribution of the dependent variable takes the form $$Y|\beta,h \sim N(X\beta,h^{-1}I).$$ I assign a conditional (uninformative) conjugate prior for $\beta$ and $h$ $$\beta|h \sim N(0,cI), h\sim G(s^{-2},v)$$ were $c\rightarrow\infty, v\rightarrow0$. It is a standard result that the marginal posterior distribution of $\beta$ is multivariate t with $$\beta|D\sim t_N (\hat{\beta},\hat{\Sigma},T).$$ What happens if $(X'X)$ is singular? In standard regression I would go for the generalized Moore-Penrose Pseudoinverse $(X'X)^+$ instead of using $(X'X)^{-1}$. However, in this case the posterior variance $\hat{\Sigma}:=c(X'X)^{-1}$ would be singular as well and I doubt that the $t$-Distribution is still well-defined. Is this correct?

And even further distracting to me: Assume I am not really interested in the posterior distribution of $\beta$ but just a linear combination $z:=A\beta$ where $A\in\mathbb{R}^{N-1 \times N}$, and $|A\hat{\Sigma}A'|\neq 0$. I would be able to sample from that distribution although its construction is based on something that is not really defined (the distribution of $\beta$). Is there a way to handle this? Or is there an essential mistake in my question that makes my whole point obsolete?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ At best, non-informative priors provide helpful results when the data uniquely identify model parameters - This observation is basically why we have ridge regression and its relatives instead of solely relying on OLS. But if the data are not sufficiently informative, typically one will either go the regularized regression route (ridge, etc) or the fully Bayes route. In the full Bayes route, just define proper, informative prior distributions over your data and the problem will be tractable. $\endgroup$
    – Sycorax ♦
    Oct 21, 2015 at 13:30
  • 1
    $\begingroup$ Thank you for your comments so far! I get the point that the posterior of $\beta$ is not properly defined. However, does this really cause problems for the random variable $z $ which is at least theoreticaly well defined? $\endgroup$
    – Stefan Voigt
    Oct 21, 2015 at 18:42
  • 1
    $\begingroup$ Well. what confuses me is that the posterior of $z$ seems plausible although the way to a solution is not satisfying at all. I am currently searching for a way to rewrite the regression equation, because I am optimistic that it would be possible to obtain directly regression parameters $z$ instead of wasting time on searching for $\beta$. However, although this seems possible in my specific case, I am still left with the question what it means if a 'bad' model is nested in a functioning one... $\endgroup$
    – Stefan Voigt
    Oct 23, 2015 at 8:47
  • $\begingroup$ If your design matrix is highly or perfectly multicollinear, it contains very little information to update your priors with, which means at least for some parameters the resulting posterior will very much resemble the prior you've started with (for more details see Poirier). If you started with an improper prior, you'll get an improper posterior. $\endgroup$
    – Durden
    Mar 18, 2023 at 14:14

1 Answer 1

Reset to default
2
$\begingroup$

The major problem with your question is that taking limits does not straightforwardly extend to measures and probability distributions. There are many different types of convergence associated with measures.

Hence, considering the conjugate $$\beta|h \sim \mathcal{N}(0,cI), h\sim \mathcal{G}(s^{-2},\nu)$$ and letting $\nu$ and $c$ go to $0$ and $\infty$, respectively, does not have a proper or unique mathematical meaning.

Now, if you consider the improper prior $$\pi(\beta,h)\propto\frac{1}{h}$$ there is no posterior distribution associated with the likelihood $$L(\beta,h|X,y)=\exp\{-h(y-X\beta)^\text{T}(y-X\beta)/2\}h^{T/2}$$ because the potential posterior does not integrate in $\beta$ conditional on $h$. There is no $$\hat{\Sigma}=(X^\text{T}X)^{-1}$$ either because the inverse does not exist and no well-defined distribution on $A\beta$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.