4
$\begingroup$

Background / motivation: I was re-listening to Prof. Strang's video lectures on linear algebra addressing the $Ax = b$ matrix-vector multiplication, and I started to realize that, although intuitive, my conceptual understanding of the properties of the typical dataset with several variables and many rows of observations (many times presented as a data.frame in R), was a bit out of focus, or imprecise.

By "typical" data set, I am referring to continuous random variables corresponding to different attributes of the object of a study - the type that could be simulated using random number generators. I am assuming no collinearity issues.

So I figured I'd post the defining properties as they apply to my favorite data set (mtcars in R), which measures different variables corresponding to technical specs (columns) for different car models (rows). Some of the numerical variables should really be factor or categorical variables, but I looked at them as numerical anyway following the result of str(mtcars): data.frame': 32 obs. of 11 variables (all of them num). For the few not acquainted with this data set:

                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
...

Linear algebra characteristics of mtcars as a matrix:

  1. Rank: Number of pivot elements in the reduced row echelon matrix (also the dimension of the column space, in point 4, below). So using the function here, and turning mtcars into a matrix: D <- as.matrix(mtcars) the function rref(D) results in the $m \times n = 11 \times 11$ identity, $I$, matrix with $32 - 11 = 21$ added rows of zeros. So there are $11$ pivots: each column has a pivot. Hence, the rank of the matrix $= 11$.

  2. Span of the column space: The $n$ vectors in the column space are independent, and their linear combinations span a subspace within $\mathbb R^{32}$ - this is the column space or $C(\text {D})$.

  3. Dimension of the column space of the matrix: All possible linear combinations of the column vectors form the column space of the matrix, $C(\text{D})$, a subspace in $\mathbb R^{32}$. The number of vectors forming any basis for this subspace is its dimension, which in this case is $11$. Alternatively, the $dim(C(\text{D})) = rank=11$.

  4. Dimension of the null space of $C(\text{D})$, symbolized as $N(\text{D})$: The basis for $N(D)$ is the number of special solutions, one for every free variable in the $\text{rref}$. The $\text{dim}\,N(D)$ is the number of columns, $n$ minus the rank of the matrix, $r$. In this case, zero. If there were any vectors, they would be the solution of $D \,x=0$, and hence, they would be in a subspace of $\mathbb R^{11}$

  5. Dimension of the row space: The row space is formed by all linear combinations of $C(\text {D'})$, within $\mathbb R^{11}$. The dimension of this subspace is $11$, because even with $32$ rows in $D$, $11$ are independent forming a basis of the space where $C(\text {D'})$ lives, i.e. $\mathbb R^{11}$ . In general, the $\text{dim}\,C(D')=rank$, as in the column space.

  6. Span of the row space: In this case the row space, $C(\text{D'})$, will span $\mathbb R^{11}$ - is a basis of $\mathbb R^{11}$, because the rank is $11$.

  7. Dimension of the null space of $C(\text{D'})$ - the 'left null space': The number of rows of $D$, which become the columns of $D'$, i.e. $m$, minus the rank of the matrix, $r$. In this case, $32 - 11 = 21$.

  8. The matrix is full rank despite $m > n$ because "full rank" is achieved if the actual rank is the largest possible for a matrix of its dimensions ($32 \times 11$), which is the lesser of the number of rows and columns ($11$).

So, having shown my personal effort in accordance to CV etiquette, the question is: Do these characteristics of the matrix of a dataset of roughly the type mentioned above (i.e. $m>>n$; continuous random variables corresponding to measurements on several attributes of the objects being studied) are complete enough? Are there more? Are the concepts accurately stated?

NOTE: I have modified the question thanks to the feedback in the comments at the risk of rendering some of them incongruous with the question as it is now, but with the intention of reducing "noise" and limiting confusion.

$\endgroup$
  • 2
    $\begingroup$ I'm not clear what you're after. What does the term "defining elements" mean? Note that the continuity of the variables doesn't really come into whether the matrix is of full column rank. e.g. if I have a column that is "exact age" (notionally continuous, so an age might be recorded as 32.178) and I have another column that is "exact age at 1 Jan 2001" (also notionally continuous), then as long as my design matrix has the constant column, those things will lead to a design that's not of full column rank, even though we have continuous variates. $\endgroup$ – Glen_b Jan 11 '16 at 3:32
  • 3
    $\begingroup$ I think your understanding is generally fine for your first two bullets. I am a bit unclear what you mean in your third bullet by "If they spanned the column space of the data". You have a full rank $32 \times 11$ matrix. Done. Your column space is spanned. Similarly in bullet 4 $m>n$ does not guarantee a full matrix. The rank is the number of linearly independent rows or columns of a full matrix. Notice that "full rank" does not equate "numerically nice" either. Hilbert matrices for example are full rank but have very high condition numbers. $\endgroup$ – usεr11852 Jan 11 '16 at 3:47
  • $\begingroup$ @Glen_b Bad choice of words - will modify... I just did. Does it make more sense now? $\endgroup$ – Antoni Parellada Jan 11 '16 at 3:59
  • $\begingroup$ @usεr11852 1. I thought the column space of this dataset would be $\mathbb R{32}$, but this is the type of semi- or mis-understandings I am trying to clarify; 2. On your second point, could it be that you overlooked the "despite" part of the sentence? Anyway, thanks for your input, and please correct the first point if its wrong in the OP. $\endgroup$ – Antoni Parellada Jan 11 '16 at 4:01
  • 2
    $\begingroup$ You point out, correctly, that the column space is not $R^{32}$ for dimensionality reasons. None the less that does not make it $R^{11}$, but some $11$ dimensional linear subspace of $R^{32}$. $\endgroup$ – Matthew Drury Jan 11 '16 at 4:40
1
$\begingroup$

I already commented on some things I believe you need to clarify further. Having said that, I think it would be good to look into the concept of the condition number of a matrix.

The condition number relates to linear dependence/multicollinearity between the columns of your matrix at hand as well as to the numerical accuracy of your estimates when using the system represented in that matrix. A matrix might be full rank but still have condition numbers that are prohibitively large to use effectively. Especially in the case of iterative algorithms where one update its systems of equations control the condition number's magnitude can make a world of difference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.