# Does the size of the design matrix change for estimation vs. predictions?

Say I have the model $$y = \beta_0 + \beta_1 x_1 + \cdots \beta_p x_p + \epsilon$$. Using $$n$$ observations of data I formed the system of equations $$\mathbf{y} = X\beta + \epsilon$$ for least squares estimation, where $$X\in M_{n\times (p+1)}$$, leading to $$\hat{\beta}_0,\hat{\beta_1}, \hat{\beta_2},...,\hat{\beta_p}$$. Denote $$\mathbf{b} = (\hat{\beta}_0,\hat{\beta_1}, \hat{\beta_2},...,\hat{\beta_p})$$.

Now I want to use my model for out of sample prediction (predicting values of $$y$$ that were not part of my data). Say I have $$k$$ values of $$y$$ I want to predict, for $$k$$ different values of the predictors $$(x_1, x_2, ... , x_p)$$.

So my estimate of $$\mathbf{y}$$ would be $$\hat{\mathbf{y}} = X\mathbf{b}$$.

So the size of $$X$$ should be $$k\times (p+1)$$, right? So are the two matrices ($$X$$ before and $$X$$ after) different? In my interpretation, I seem to see them used interchangeably in notes and online etc. (Also the $$\mathbf{y}$$ should be a different size too?)

Is it common to have the number of observations of data and the number of predictions of $$\mathbf{y}$$ to be the same? Have I misunderstood something?

For Example:

Let me use $$X^{(1)}$$ as the matrix used for estimation above, and $$X^{(2)}$$ be the matrix used for prediction below.

Then, for instance, the hat matrix $$H$$ such that $$\mathbf{\hat{y}}= H \mathbf{y}$$ is often given denoted as $$H =X\underbrace{(X^T X)^{-1}X^T}_{\text{used to estimate }\mathbf{b}}$$. Should this "strictly speaking" be $$H =X^{(2)}\underbrace{((X^{(1)})^T X^{(1)})^{-1}(X^{(1)})^T}_{\text{used to estimate }\mathbf{b}}$$ instead? And properties such as $$HX = X$$, this would only be true as $$HX^{(1)} = X^{(2)}$$?

There is no particular reason why the number of predictions should be limited by the number of training data points. In fact, the whole point of the regression is to create a predictor for an infinite (!!) number of points in the region spanned by variables $$(x_1, x_2,..., x_p)$$. In other words, $$X$$ before and $$X$$ after are not necessarily the same and may have different number of rows.
However, your example likely appears in the context of predicting the outcomes for the predictor variable values in the data set (which are not the same as the observed values), so all the matrices there are $$X^{(1)}$$.
• Do you mean an in-sample prediction? That is, make a model using the data set ($X^{(1)}$ and observed $\mathbf{y}$) we get, and then predict those observed values using the same data set? (Check $X\mathbf{b}$ when $X=X^{(1)}$?) – user523384 Nov 19 '19 at 10:08
• You may call it in-sample prediction... Here the actual outcomes $\mathbf{y}$ and the corresponding estimates $\mathbf{\hat{y}}$ are for the same predictor variables $X^{(1)}$. Your mathematical manipulations are correct, but it doesn't make much sense to hide the new predictor variables, used for the new estimates, in matrix $H$. – Roger Vadim Nov 19 '19 at 10:13