Textbook literature often denotes the estimated weight matrix of a linear regression model $y = Wx + \epsilon, \epsilon \sim \mathcal{N}(0,\sigma^2)$ by $\hat W$ due to the inherent variability in the sample size. Limited access to all possible instances hinders our ability to find the true matrix $W$. Additionally, the relationship between the inputs ($x$) and the output ($y$) might not be linear, resulting in uncertainty represented by the random variable $\epsilon$. Incorporating both sources of randomness into the estimate of $W$, we can see that $\epsilon$ is random $\rightarrow y$ is random $\rightarrow \hat W$ is random. Therefore, $\hat W | data \sim \mathcal{N}(W,\sigma^2(X^TX)^{-1})$, where $X \in \mathbb{R}^{n \times p}$, and $p$ is the number of features while $n$ is the sample size.

  1. Do you agree with me that in this model, we have two sources of randomness/uncertainty: the sample size variation and the uncertainty induced by the input-output relationship?
  2. As $n \rightarrow \infty$, $(X^TX)^{-1}$ approaches the true population inverse covariance matrix of the feature vector, if the features are naturally random. However, if the features are deterministic (hence the randomness of the input is due to the sample size, not to inherent randomness in the input), then I guess each element of the covariance matrix of the estimate of $W$ should go to zero, indicating that the uncertainty about the model weight matrix becomes negligible. However, I think it will converge to constant values, but what do those values mean? In this case, I think the model projects spurious randomness, and this should be appropriately treated.
  3. If the distribution of $\hat W$ accounts for all sources of randomness, Why do we need a Bayesian approach to estimate the weight then? There is already a way to estimate the uncertainty when predicting a new instance: $p(y_{test} | Data) = \int p(y_{test} | \hat W,x_{test})p(\hat W|data) d \hat W$, where $p(\hat W|data)\sim \mathcal{N}(W,\sigma^2(X^TX)^{-1}$) (assuming that I do have the model $p(y_{test} | \hat W,x_{test})$. I understand that the original assumption in this modelling is that we have true value of $W$ which is the true mean of the distribution of $\hat W$ and based on this assumption, we should only use that value. However, I am a bit confused of why we can't model the uncertainty this way.

1 Answer 1

  1. Notice that the sample covariance is $S_n = \frac{1}{n}\sum_i x_i x_i^T=\frac{1}{n}X^TX$. If the features are drawn from a distribution with covariance matrix $\Sigma$, then $S_n$ converges in probability to $\Sigma$ when $n$ approaches infinity. In other words, $(X^TX)^{-1} \to n^{-1} \Sigma^{-1} \to 0$ when $n \to \infty$.

  2. In linear regression the features are usually considered as fixed. In particular, the distribution of the estimator $\hat W \sim \mathcal N(W,\sigma^2(X^TX)^{-1})$ is due to the randomness of $y$ only. (Intuitively, it is the distribution you will get by sampling $y$ over and over with the same features). If you would like to consider how $\hat W$ varies due to randomness of the features, you will have to additionally model the distribution of $x$, which will result in a much more complicated distribution of $\hat W$ (and probably impossible to find in closed form).

  3. The estimator $\hat W$ is fixed by the data, so the expression $p(\hat W|data)$ doesn't make a lot of sense (it is just equal to 1). To make sense of it you would have to treat $W$ itself as the random variable, which is exactly the Bayesian approach.

  • $\begingroup$ I think the sample covariance is divided by $\frac{1}{n-1}$ not $\frac{1}{n}$. Does your statement regarding (2) also hold for the randomness of the features due to the varying sampling size? I haven't caught on why $p(\hat W | data)$ equals to 1. $\endgroup$
    – rando
    Commented May 9, 2023 at 16:44
  • $\begingroup$ That's a matter of convention, which doesn't matter when $n \to \infty$. Yes (2) also holds when $n$ is random. $\hat W$ is a deterministic function of the data. Once you know the data, there is no more uncertainty about the value of $\hat W$. $\endgroup$
    – J. Delaney
    Commented May 9, 2023 at 17:14
  • $\begingroup$ Thanks. Regarding the last sentence, note that we ended up with $P(\hat W)$ even though we have $data$ as the relationship between $x$ and $y$ is deterministic. I am referring to $\epsilon$ which causes $\hat W$ to have a distribution. $\endgroup$
    – rando
    Commented May 9, 2023 at 17:20
  • 1
    $\begingroup$ $W$ is not a function of $\epsilon$, so it doesn't. Furthermore, after the data is observed, $\epsilon$ is fixed (otherwise every time you looked at $Y$ the values would change), which is why $\widehat{W}$ is only random up until you observe the data. Your lack of understanding is something most of us had to work through at one time or another, no excuses needed!!! :) $\endgroup$
    – jbowman
    Commented May 11, 2023 at 23:46
  • 1
    $\begingroup$ @jbowman Oh, I see what you and J. Delaney meant. I failed to see that $\epsilon$ is fixed when data is given and hence there is no stochasticity. $\endgroup$
    – rando
    Commented May 12, 2023 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.