I'm curious about how are the loading in a simple confirmatory factor analysis determined mathematically. Also, the intuition is also hazy to me as to how are the factor loads determined when a variable which is not observed is present in the system and still its impact is being estimated on the observed variables (The factor loading)
The basic concept is as follows:
Determine what the covariance matrix should be, as a function of the factor model and the unknown loadings.
Calculate the observed variance/covariance matrix from the data.
Estimate the parameters. Different software uses different algorithms. A conceptually simple choice is a least squares fit. Choose the parameters to make the theoretical matrix close to the observed matrix.
For a simple, one factor model, the covariance of $X_i, X_j$ is $\lambda_i \times \lambda_j$ and the variance of $X_i$ is $\lambda_i^2 + \sigma_i^2$
If I have $p$ variables loading on the factor, that gives me $2p$ parameters from the model and $p(p+1)/2$ observed entries in the covariance matrix. Note that I am fixing the variance of the latent factor to 1. Alternatively, I can fix one of the loadings to 1 and estimate the latent factor variance.
Note that if $p=2$ I have too many parameters and if $p=3$, the model is saturated and I can choose parameters to give a perfect fit. You want at least 4 observed variables to make this interesting.