# Why does it matter that bases are linearly independent?

I am taking an advanced linear algebra course and am once again confused by a lot of the concepts. I understand that the definition of a basis is a set of vectors that spans the vector space and is linearly independent. Can anyone provide any intuition on why this matters?

I can visualize (kind of) what it means for a set to span a vector space, and I understand (from regression analysis) why it is important that vectors are linearly independent, but I don't really understand why it matters if they are both, ie why it matters if they form a basis.

For reference I am an undergraduate interested in statistics and data analysis. I have taken courses in mathematical statistics, regression analysis and am currently enrolled in time series analysis. I am somewhat familiar with PCA. Any intuition that can be provided through any of those lenses would be much appreciated.

• If you have a set of $n$ vectors that is not linearly independent, then that means you can form at least one of them as a linear combination of the others. The span is just the set of all linear combinations of those vectors, so if one of the vectors can be written as a linear combination of the others, then (by substitution), any vector in the span can be written as a linear combination of the other $n-1$ vectors. So you have at least one vector in this set of vectors that is 'redundant'. A 'basis' is defined to exclude this type of redundancy. – Reinstate Monica Jan 25 '18 at 23:51
• Just as travel and learning new languages broadens one's mind by offering alternatives to the "obvious" ways of thinking and viewing the world one grows up with, learning linear algebra a little more generally and abstractly helps one appreciate issues like this one. Consider generalizing your concepts of spanning and linear independence to vectors and scalars that are required to be integers. Suddenly those concepts are no longer so intimately linked. – whuber Jan 26 '18 at 0:33

Perhaps a little motivation might help.

Suppose you have a dataset with $10$ features. Visualizing them is a bit challenging. However, if you are told that actually all data set live in a $2$ dimensional subspace and it is not a straight line. It would be great if you can visualize this data set by projecting your data onto this $2$ dimensional subspace.

A basis is a convenient tool for this. How many vectors are needed in the basis? It is necessary for us to have $2$ directions to describe this subspace. Can you describe any vector in this two dimensional subspace? Answer would be yes since the basis of the subspace spans the subspace.

In particular notice that we can represent an arbitrary vector as a unique linear combination of the vectors in the subspace. It can be represented as a basis span the subspace and the uniqueness is due to the linearly independence property. Hence, a basis provide us with a well-defined coordinate system.

Of course, in practice, we rarely would have all data set live in $2$ dimensional subspace but perhaps we can find a lower dimensional subspace to describe this data set by projecting them into the subspace and a basis would provide us with a convenient coordinate system.

• Okay, I think I understand. So we need linear independence because it means that whichever "arbitrary vector" we create is a UNIQUE linear combination of our basis vector? – agra94 Jan 25 '18 at 23:43
• yup, that is one of the property. It is the largest independent set and the smallest spanning set (if it is finite). – Siong Thye Goh Jan 26 '18 at 2:46

Let $\{ \boldsymbol{x}_1, ..., \boldsymbol{x}_n \}$ be a set of vectors with $\boldsymbol{x}_i \in \mathbb{R}^m$, and suppose that they are not linearly independent. This means that any of these vectors can be written as a linear combination of the others. Without loss of generality, let us suppose that:

$$\boldsymbol{x}_n = \sum_{i=1}^{n-1} \alpha_i \boldsymbol{x}_i.$$

Then the set of vectors has span:

\begin{equation} \begin{aligned} \text{Span}(\boldsymbol{x}_1, ..., \boldsymbol{x}_n) &\equiv \{ \boldsymbol{v} \in \mathbb{R}^m | (\exists \boldsymbol{\beta} \in \mathbb{R}^n): \boldsymbol{v} = \sum_{i=1}^{n} \beta_i \boldsymbol{x}_i \} \\ &= \{ \boldsymbol{v} \in \mathbb{R}^m | (\exists \boldsymbol{\beta} \in \mathbb{R}^n): \boldsymbol{v} = \sum_{i=1}^{n-1} (\beta_i + \beta_n \alpha_i) \boldsymbol{x}_i \} \\ &= \{ \boldsymbol{v} \in \mathbb{R}^m | (\exists \boldsymbol{\beta}^* \in \mathbb{R}^{n-1}): \boldsymbol{v} = \sum_{i=1}^{n-1} \beta_i^* \boldsymbol{x}_i \} \\ &= \text{Span}(\boldsymbol{x}_1, ..., \boldsymbol{x}_{n-1}). \end{aligned} \end{equation}

This tells us that if the vectors are not linearly independent, it is possible to remove one of the vectors without reducing the span (i.e., there is some redundancy in the set of vectors). This can continue until you are left with a set of linearly independent vectors with the same span as the original set.

It is often useful in linear algebra to assume that this process of removal of redundant vectors is complete, so you are left with a linearly independent set of vectors with a particular span. Hence, the basis is defined to require the condition of linear independence.

The answers already given are great, but I think one example that you may come across while analyzing data is the following.

You are asked to analyze the relationship between the salary $y$, the age $x_1$, the number of years spent in the company $x_2$, and the total number of working years $x_3$. Now your data points live in a three dimensional space and, after estimation, you obtain a unique equation :

$$y=1000+100x_1+300x_2+200x_3$$

You could reach a conclusion such as "the best way to increase your salary in this company is to stay longer in the company".

Now, say you decide to build a feature (or, even better, someone did it before giving you the dataset which is the number of non working years : $x_4= x_1-x_2-x_3$ ).

And you want to produce a linear model from this data. Now the equation could be (most packages or library would simply crash if you fed them such data) :

$$y=1000+100x_1+300x_2+200(x_1-x_2-x_4)$$

or

$$y=1000+100(x_2+x_4+x_3)+300x_2+200x_3$$

Though the equations would produce the same predictions, you could not infer the importance of the parameters and their impact on the value you are trying to predict.