Data space, variable space, observation space, model space (e.g. in linear regression)

Question

Suppose we have the data matrix $\mathbf{X}$, which is $n$-by-$p$, and the label vector $Y$, which is $n$-by-one. Here, each row of the matrix is an observation, and each column corresponds to a dimension/variable. (assume $n>p$)

Then what do data space, variable space, observation space, model space mean?

Is the space spanned by the column vector, a (degenerated) $n$-D space since it has $n$ coordinates while being rank $p$, called variable space since it is spanned by variable-vector? Or is it called observation space since each dimension/coordinate corresponds to an observation?

And what about the space spanned by the row vectors?

Answer 1

These terms appear in some books on multivariate statistics. Suppose you have $n$ individuals by $p$ quantitative features data matrix. Then you can plot individuals as points in the space where the axes are the features. That will be classic scatterplot, aka variable space plot. We say, the cloud of individuals span the space defined by the axes-features.

You could as well conceive of the scatterplot with points being the variables and the axes being the individuals. Absolutely like previous, only topsy-turvy. That will be subject space plot (or observation space plot) with the variables spanning it, the individuals defining it.

Note that if (as often) $n>p$ then, in the second case, only some $p$ dimensions out the $n$ dimensions are nonredundant; that means that you can and may draw the $p$ variable points on $p$-dimensional plot $^1$. Also, by tradition variable points are usually connected with the origin and so they appear as vectors (arrows). We use subject space representation mostly to show relationships between variables, therefore we drop the axes-subjects and depict points as arrows, for convenience.

If features (columns of the data matrix) were centered before drawing the subject space plot then the cosines of the angles between the variable vectors are equal to their Pearson correlations, while the vector lengths are equal to the variables' norms (root sum of squares) or standard deviations (if divided by the df).

Variable space and subject space are two sides of the same coin, they are the same Euclidean analytic space, only presented mirrorlike to each other. They share the same properties, such as the nonzero eigenvalues and eigenvectors. It is possible therefore to plot both subjects and variables side by side as points in the space of the principal axes (or other orthogonal basis) of that analytic space, - this joint plot is called biplot. I don't know exactly what term "data space" mean - if it means something specific then I suppose it is that common analytic space of which subject space and variable space are the two hypostases.

Some local links:

• Pictures showing subject space representation of principal components (PCA) vs linear regression (compare that with traditional, variable space (scatterplot) representation of regression and PCA), factor analysis vs PCA (compare that with variable space representation of factor analysis vs PCA, again regression, partial correlation, correlation, regressional b vs correlation, covarianse vs common variance, suppressor.
• Theoretical explanation of biplot. One self-study explaining structure of biplot in PCA.
• See also a post trying to figure out if one can geometrically solve PCA task on the subject space plot (it appears that the PCs define the ellipse; but how to find that unique ellipse?).

---

$^1$ Imagine that you have n=5 individuals and p=2 variables and you somehow managed magically to draw the 2 points in the 5-dimensional space. Then you may rotate the subspace defined by any 2 of the axes in such a way that it embeds the 2 points (which thus span that plane from now on); after it, you drop safely the other 3 axes (dimensions) since they have become unnecessary. The position of the two variable points relative each other was preserved.