# Data space, variable space, observation space, model space (e.g. in linear regression)

Suppose we have the data matrix $\mathbf{X}$, which is $n$-by-$p$, and the label vector $Y$, which is $n$-by-one. Here, each row of the matrix is an observation, and each column corresponds to a dimension/variable. (assume $n>p$)

Then what do data space, variable space, observation space, model space mean?

Is the space spanned by the column vector, a (degenerated) $n$-D space since it has $n$ coordinates while being rank $p$, called variable space since it is spanned by variable-vector? Or is it called observation space since each dimension/coordinate corresponds to an observation?

And what about the space spanned by the row vectors?

• These are not universally-known terms. Do you have a reference? If not, we might be guessing at what they are intended to mean. – whuber Jan 26 '16 at 23:29
• I do not have a reference. I once heard my professor told it some time ago. – user3813057 Jan 26 '16 at 23:37
• I am pretty sure, then, that your professor defined these terms at some point. Maybe they are in your class notes... . – whuber Jan 26 '16 at 23:38

These terms appear in some books on multivariate statistics. Suppose you have n individuals by p quantitative features data matrix. Then you can plot individuals as points in the space where the axes are the features. That will be classic scatterplot, aka variable space plot. We say, the cloud of individuals span the space defined by the axes-features.

You could as well conceive of the scatterplot with points being the variables and the axes being the individuals. Absolutely like previous, only topsy-turvy. That will be subject space plot (or observation space plot) with the variables spanning it, the individuals defining it.

Note that if (as often) n>p then, in the second case, only some p dimensions out the n dimensions are nonredundant; that means that you can and may draw the p variable points on p-dimensional plot $^1$. Also, by tradition variable points are usually connected with the origin and so they appear as vectors (arrows). We use subject space representation mostly to show relationships between variables, therefore we drop the axes-subjects and depict points as arrows, for convenience.

If features (columns of the data matrix) were centered before drawing the subject space plot then the cosines of the angles between the variable vectors are equal to their Pearson correlations, while the vector lengths are equal to the variables' norms (root sum of squares) or standard deviations (if divided by the df).

Variable space and subject space are two sides of the same coin, they are the same Euclidean analytic space, only presented mirrorlike to each other. They share the same properties, such as the nonzero eigenvalues and eigenvectors. It is possible therefore to plot both subjects and variables side by side as points in the space of the principal axes (or other orthogonal basis) of that analytic space, - this joint plot is called biplot. I don't know exactly what term "data space" mean - if it means something specific then I suppose it is that common analytic space of which subject space and variable space are the two hypostases.

$^1$ Imagine that you have n=5 individuals and p=2 variables and you somehow managed magically to draw the 2 points in the 5-dimensional space. Then you may rotate the subspace defined by any 2 of the axes in such a way that it embeds the 2 points (which thus span that plane from now on); after it, you drop safely the other 3 axes (dimensions) since they have become unnecessary. The position of the two variable points relative each other was preserved.

• +1. But I am not sure what is the exact mathematical meaning of saying that the variable and the subject space are "the same Euclidean analytic space". – amoeba says Reinstate Monica Jan 27 '16 at 16:39
• @amoeba, Without being mathematical in the answer I hoped that it is intuitively transparent (especially for linear algebra experts like you). For instance, in the singular value decomposition of the data matrix (svd, on which biplot is based) - what space do the eigenvalues and the left and right eigenvectors characterize? Isn't it the same analytic space, which could be laid out various ways,- among which are (i) rows and columns as points by principal axes as axes; (ii) rows as points by columns as axes; (iii) columns as points by rows as axes? – ttnphns Jan 27 '16 at 17:16