Say I have the model $y = \beta_0 + \beta_1 x_1 + \cdots \beta_p x_p + \epsilon $. Using $n$ observations of data I formed the system of equations $\mathbf{y} = X\beta + \epsilon$ for least squares estimation, where $X\in M_{n\times (p+1)}$, leading to $\hat{\beta}_0,\hat{\beta_1}, \hat{\beta_2},...,\hat{\beta_p} $. Denote $\mathbf{b} = (\hat{\beta}_0,\hat{\beta_1}, \hat{\beta_2},...,\hat{\beta_p})$.

Now I want to use my model for out of sample prediction (predicting values of $y$ that were not part of my data). Say I have $k$ values of $y$ I want to predict, for $k$ different values of the predictors $(x_1, x_2, ... , x_p)$.

So my estimate of $\mathbf{y}$ would be $\hat{\mathbf{y}} = X\mathbf{b}$.

So the size of $X$ should be $k\times (p+1)$, right? So are the two matrices ($X$ before and $X$ after) different? In my interpretation, I seem to see them used interchangeably in notes and online etc. (Also the $\mathbf{y}$ should be a different size too?)

Is it common to have the number of observations of data and the number of predictions of $\mathbf{y}$ to be the same? Have I misunderstood something?

For Example:

Let me use $X^{(1)}$ as the matrix used for estimation above, and $X^{(2)}$ be the matrix used for prediction below.

Then, for instance, the hat matrix $H$ such that $\mathbf{\hat{y}}= H \mathbf{y}$ is often given denoted as $H =X\underbrace{(X^T X)^{-1}X^T}_{\text{used to estimate }\mathbf{b}}$. Should this "strictly speaking" be $$H =X^{(2)}\underbrace{((X^{(1)})^T X^{(1)})^{-1}(X^{(1)})^T}_{\text{used to estimate }\mathbf{b}}$$ instead? And properties such as $HX = X$, this would only be true as $HX^{(1)} = X^{(2)}$?


There is no particular reason why the number of predictions should be limited by the number of training data points. In fact, the whole point of the regression is to create a predictor for an infinite (!!) number of points in the region spanned by variables $(x_1, x_2,..., x_p)$. In other words, $X$ before and $X$ after are not necessarily the same and may have different number of rows.

However, your example likely appears in the context of predicting the outcomes for the predictor variable values in the data set (which are not the same as the observed values), so all the matrices there are $X^{(1)}$.

  • $\begingroup$ Thank your for your response! That makes a lot of sense! (I never thought of the idea of being a predictor for the span of the variables, that makes it a lot clearer) Could you please elaborate on what you mean by your second paragraph? I think it would clear up a lot but i didn't quite pick up what you meant $\endgroup$ – user523384 Nov 19 '19 at 9:59
  • $\begingroup$ Do you mean an in-sample prediction? That is, make a model using the data set ($X^{(1)}$ and observed $\mathbf{y}$) we get, and then predict those observed values using the same data set? (Check $X\mathbf{b}$ when $X=X^{(1)}$?) $\endgroup$ – user523384 Nov 19 '19 at 10:08
  • $\begingroup$ You may call it in-sample prediction... Here the actual outcomes $\mathbf{y}$ and the corresponding estimates $\mathbf{\hat{y}}$ are for the same predictor variables $X^{(1)}$. Your mathematical manipulations are correct, but it doesn't make much sense to hide the new predictor variables, used for the new estimates, in matrix $H$. $\endgroup$ – Roger Vadim Nov 19 '19 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.