$k$ is the dimensionality of the spline basis expansion of 1 or possibly more covariates. By default (with the thinplate spline basis and a spline for a single covariate) the basis will contain $k$ = k - 1 basis functions. These functions describe a function space. (This space is a vector space.)
A basis is what results from a basis expansion. In other words, a basis is the set of functions, and the individual functions might be called basis functions.
The dimension of the basis is typically written as $n$, the number of basis functions in the basis, but in mgcv typically this is written as $k$.
A layperson's explanation of $k$ is simply that $k$ is the maximum possible degrees of freedom allowed for a smooth term in the model. Note that $k$ is the maximum degrees of freedom allowed for a single smooth term in the model, but it invariably will not be k. Typically $k$ will be k - 1 due to the identifiability constraint on the smooth term.
The terms you refer to from linear algebra are the same as the terms basis and dimension for splines in GAMs.
A vector in a vector space is a linear combination of the bases of the vector space (or basis):
$$\mathbf{v} = a_1\mathbf{b}_{i_1} + a_2\mathbf{b}_{i_2} + \cdots + a_n\mathbf{b}_{i_n}$$
where the a are scale weights specifying how much of each basis contributes to the vector $\mathbf{v}$. The $\mathbf{b}$ are the vectors that are contained in the vector space. The basis is the collection of vectors $\mathbf{b}$, and the dimension of the basis is $n$, the number of vectors in the basis.
In a spline basis we have the same thing but we might think of it more as a linear combination of basis functions but it is conceptually the same
$$\mathbf{f} = a_1\mathbf{b}_{i_1} + a_2\mathbf{b}_{i_2} + \cdots + a_n\mathbf{b}_{i_n}$$
where now $\mathbf{b_{._n}}$ are the values of the basis function evaluated at a given value of the covariate
$$f(x_i) = a_1b_1(x_i) + a_2b_2(x_i) + \cdots + a_nb_n(x_i)$$
(I may be butchering notation here, but...) hence we are evaluating the function $f$, the spline, at the $i$th value of the covariate $x$. The value taken by the function is a linear combination of the values of the $n$ basis functions, each evaluated at the $i$th value of the covariate. The scalars $a$ are the coefficients estimated when fitting the GAM.
