What is a "factor" in factor analysis? What is a factor from a linear algebra point of view?  Is it a vector, matrix, basis, tuple, coordinate system or something else?
 A: The usual factor analysis model is 
$$\mathbf{Y} = \mathbf{\mu}+ \mathbf{\Phi}\mathbf{L} + \mathbf{\eta},$$
where $\mathbf{Y}$ represents a collection of $n$ observations of $k$ random variables; i.e. it is a matrix of $n \times k$ size. $\mathbf{\mu} = \mathbf{1}_n^{\prime} (\mu_1, \mu_2, \ldots, \mu_k)$
   is also an $n\times k$ matrix, constant in each column, giving the
   means of the $k$ variables. $\mathbf{\Phi}$ is a $n\times p$ matrix of $p \le k$ factors; $\mathbf{L}$ is an $p \times n$ matrix of (unknown) constants (to be estimated); and $\mathbf{\eta}$ is an $n \times k$ matrix of errors.  The rows of $\mathbf{\eta}$ are independent and independent of $\Phi$. The elements within row $i$ have mean $0$ and variance $\sigma_i^2$.  The quantities on the right hand side are unobservable but (usually) fewer in number than the $nk$ data values, and so are (up to a degree of ambiguity discussed below) identifiable. Note that $\eta$ is not to be identified, but only its row variances $\sigma_i^2$, called "uniquenesses".
In the language of factor analysis, the factors are the columns of $\mathbf{\Phi}$. From $k$ original variables it extracts $p\lt k$ factors. One can say that a "factor" is a whole column; i.e., a collection of $n$ realizations of a random variable, or, more abstractly, a random variable itself.  Usually it is assumed that the factors are uncorrelated and standardized, i.e. have unit variance.
The rows of $\mathbf{L}$ are called factor loadings.
Note that this model is unique only up to orthogonal transformations, in that 
$$\mathbf{Y} = \mathbf{\mu}+ (\mathbf{\Phi P^\top})(\mathbf{PL}) + \mathbf{\eta},$$
where $\mathbf{P}$ is any orthogonal matrix.
A: Factor is a vector. The set of factors give you a coordinate system, a basis. Factor loadings are sets of coordinates in this basis.
Let's say you have a $T\times n$ matrix $X=x_{ti}$. Imagine that it's a path of particle in n-dimensional space, where $t$ is time, and $i$ is the dimension. 
What factor analysis does is simply change the coordinate system from your current basis to something else, then your $X$ matrix becomes a $T\times n$ matrix $A=a_{ti}$. It's the same path in time, except in different coordinates. The actual coordinates of a point at time $t$ are called loadings or a score in PCA, i.e. each row is a particular point in time, a loading.
The reason to transform the coordinates is usually convenience or clarity. For instance, these would be coordinates of circular motion in cartesian system (x,y):
    0.8415    0.5403
    0.9093   -0.4161
    0.1411   -0.9900
   -0.7568   -0.6536
   -0.9589    0.2837
   -0.2794    0.9602
    0.6570    0.7539
    0.9894   -0.1455
    0.4121   -0.9111
   -0.5440   -0.8391

Here's the same in polar system (angle,radius):
1.0000    1.0000
2.0000    1.0000
3.0000    1.0000
4.0000    1.0000
5.0000    1.0000
6.0000    1.0000
0.7168    1.0000
1.7168    1.0000
2.7168    1.0000
3.7168    1.0000

The polar system is obviously suited better for this process, as you can shrink the dimensionality of the system. It's essentially one-dimensional motion along the circle's circumference.
Factor analysis is usually linear in some way, and doesn't do cool stuff like this, but still works for many processes.
