# Structural equation modeling mathematical/analytical estimation of latent variables

I want to know how latent variables are estimated in structural equation modeling. I do not understand at what point and in what way the standard SEM software comes up with the estimates for latent variable variances/covariances etc.

To demonstrate, I have simple 3-wave intercept-only panel model based on

$$x_{it} = \alpha_{i} + \delta_{it}; \ i = 1,...,n; \ t = 1,2,3$$,

the model is trivial, I just chose it because it is the simplest example I could think of. I am uninterested in the means/intercepts, so assume grand mean centered variables.

In structural equation modeling, we achive this by specifying a measurement and structural part of the model on wide-format data (this is the basis for fixed- and random-effects models in SEM, as outlined in Bollen & Brand 2010, but here I am using an alternative notation outlined in Bollen 1989, p.395)

$$\eta^* = B\eta^* + \zeta^* \\ y^* = \Lambda \eta^*$$

where

$$\eta^* = (\alpha, x_1, x_2, x_3)$$, $$\zeta^* = (\alpha, \delta_1, \delta_2, \delta_3)$$, $$y^* = (x_1, x_2, x_3)$$

$$\Lambda = \left[\begin{array}{l}0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{array}\right]$$, $$B = \left[\begin{array}{l}0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0\end{array}\right]$$, $$\Psi = \left[\begin{array}{l}\alpha^{2} & 0 & 0 & 0 \\ 0 & \delta_{1}^{2} & 0 & 0 \\ 0 & 0 & \delta_{2}^{2} & 0 \\ 0 & 0 & 0 & \delta_{3}^{2}\end{array}\right]$$.

The model-implied covariance matrix is achieved by putting the equation for $$\eta^*$$ in reduced-form and working out $$y^*y^{*\intercal}$$, which (if we respect the assumptions reflected in $$\Psi$$, i.e., no error covariances etc.) leads to

\begin{align}\Sigma(\theta) & = \Lambda(I - B)^{-1}\Psi(I - B)^{-1 \intercal}\Lambda^{\intercal} \\ & = \left[\begin{array}{c} \alpha^{2} + \delta_{1}^{2} & & \\ \alpha^{2} & \alpha^{2} + \delta_{2}^{2} & \\ \alpha^{2} & \alpha^{2} & \alpha^{2} + \delta_{3}^{2} \end{array}\right] \end{align}

I understand up to here, and the results are what we should expect: the observed variable at each point in time has just been decomposed into a time-invariant and time-varying error part.

My questions are:

1) What are the parameters in $$\theta$$, given that the factor loadings/regression coefficients in $$B$$ have been set to 1.0? Aren't they $$\alpha^{2}, \delta_{t}^{2}$$?

2) How do I solve for $$\alpha^{2}, \delta_{t}^{2}$$? Do I set the equation equal to the observed covariance matrix $$\Sigma(\theta) = S$$, work out the partial derivative with respect to each of the unknowns, set to zero and solve?

• You're asking a lot of questions here, and it's not easy to follow. You start off talking about SEM, and then start to talk about time invariance - is this longitudinal SEM? A three wave model with a single variable is just identified, so the SEM isn't very interesting. Alpha and delta are not latent variables (they are eta and ksi - sorry, not sure how to put greek letters in comments). The latent variables are not estimated - their variance is estimated, and their mean/intercept is estimated, but the variable doesn't need to exist. – Jeremy Miles Nov 28 '19 at 6:02
• Try asking more specific questions, and someone is more likely to be able to help. I get lost in this one. – Jeremy Miles Nov 28 '19 at 6:03
• Thank you for the feedback/comments, Jeremy. I have tried to streamline the question, added some possibly important information and have tried to be more specific about my questions. – hendogg87 Nov 28 '19 at 13:50

## 2 Answers

Thanks for the help Jeremy. I found the solution in Bollen (1989), p. 111.

Assume again the three-wave version of the model above, $$x_{it} = \alpha_i + \delta_{it}$$ and, for the sake of demonstration, assume the observed covariance matrix $$S$$ is

\begin{align} S & = \begin{bmatrix} x_{1}^{2} & & \\ x_{2}x_{1} & x_{2}^{2} & \\ x_{3}x_{1} & x_{3}x_{2} & x_{3}^{2} \end{bmatrix} \\ & = \begin{bmatrix} 2.106 & & \\ 1.079 & 1.958 & \\ 1.108 & 1.017 & 2.111 \end{bmatrix} \end{align}

the model-implied covariance matrix is

\begin{align} \Sigma(\hat{\theta}) & = \begin{bmatrix} \hat{\phi} + \hat{\psi}_{1} & & \\ \hat{\phi} & \hat{\phi} + \hat{\psi}_{2} & \\ \hat{\phi} & \hat{\phi} & \hat{\phi} + \hat{\psi}_{3} \end{bmatrix} \end{align}

where $$\hat{\phi} = var(\alpha)$$ and $$\hat{\psi}_{t} = var(\delta_t)$$.

We can work out the variance of $$\alpha$$ with unweighted least squares,

\begin{align} F_{ULS} & = \frac{1}{2}tr\big((S - \Sigma(\hat{\theta}))^{2}\big) \\ & = \frac{1}{2}tr((x_{1}^{2} - \hat{\phi} + \hat{\psi}_{1})^{2} + (x_{1}x_{2} - \hat{\phi})^{2} + (x_{1}x_{3} - \hat{\phi})^{2} + (x_{2}^{2} - \hat{\phi} + \hat{\psi}_{2})^{2} + (x_{2}x_{3} - \hat{\phi})^{2} + (x_{3}^{2} - \hat{\phi} + \hat{\psi}_{3})^{2}). \end{align}

We take the first derivative with respect to $$\hat{\phi}$$ which works out to

\begin{align} \frac{\partial{F_{ULS}}}{\partial{\hat{\phi}}} & = -x_{1}^{2} + \hat{\phi} + \hat{\psi}_{1} -x_{1}x_{2} + \hat{\phi} -x_{1}x_{3} + \hat{\phi} -x_{2}^{2} + \hat{\phi} + \hat{\psi}_{2} -x_{2}x_{3} + \hat{\phi} -x_{3}^{2} + \hat{\phi} + \hat{\psi}_{3}. \end{align}

Use covariance algebra together with the model assumptions to show

\begin{align} x_{t}^{2} & = (\alpha + \delta_{t})^{2} \\ & = (\alpha + \delta_{t})(\alpha + \delta_{t}) \\ & = \alpha^{2} + 2\alpha\delta_{t} + \delta_{t}^{2} \\ & = \alpha^{2} + \delta_{t}^{2} \\ & = \phi + \psi_{t} \end{align}

so $$\psi_{t} = x_{t}^{2} - \phi$$, which can be substituted into the derivative above

\begin{align} \frac{\partial{F_{ULS}}}{\partial{\hat{\phi}}} & = -x_{1}^{2} + \hat{\phi} + (x_{1}^{2} - \hat{\phi}) -x_{1}x_{2} + \hat{\phi} -x_{1}x_{3} + \hat{\phi} -x_{2}^{2} + \hat{\phi} + (x_{2}^{2} - \hat{\phi}) -x_{2}x_{3} + \hat{\phi} -x_{3}^{2} + \hat{\phi} + (x_{3}^{2} - \hat{\phi}). \end{align}

which simplifes to

\begin{align} \frac{\partial{F_{ULS}}}{\partial{\hat{\phi}}} & = -x_{1}x_{2} + \hat{\phi} -x_{1}x_{3} + \hat{\phi} -x_{2}x_{3} + \hat{\phi}. \end{align}

We set this to zero and solve for $$\hat{\phi}$$, the variance of the latent variable $$\alpha$$

\begin{align} 0 & = -x_{1}x_{2} + \hat{\phi} -x_{1}x_{3} + \hat{\phi} -x_{2}x_{3} + \hat{\phi} \\ 3\hat{\phi} & = x_{1}x_{2} + x_{1}x_{3} + x_{2}x_{3} \\ 3\hat{\phi} & = 1.079 + 1.108 + 1.017 \\ \hat{\phi} & = \frac{1.079 + 1.108 + 1.017}{3} \\ \hat{\phi} & = 1.068. \end{align}

Which is just the average covariance on the off-diagonal. I haven't tried this for the ML method, but I assume it should be straightforward. The results can also be confirmed in lavaan, see code below.

# Packages
install.packages("lavaan")
library(lavaan)
# Observed covariance matrix
S <- matrix(c(2.106, 1.079, 1.108,
1.079, 1.958, 1.017,
1.108, 1.017, 2.111), nrow = 3, ncol = 3)
# Names
colnames(S) <- c("x1", "x2", "x3")
rownames(S) <- c("x1", "x2", "x3")
# Lavaan model
m1 <- '
a =~ 1*x1 + 1*x2 + 1*x3
'
m1.fit <- sem(m1, sample.cov = S, sample.nobs = 1000)
summary( m1.fit)

• Thanks for posting! I understand now. :) – Jeremy Miles Dec 2 '19 at 17:39
• Typical somehow: have to know the answer before you can properly ask the question! Thanks for pointing me in the right direction! – hendogg87 Dec 2 '19 at 18:06

Your model isn't clear to me, that's not necessarily bad, but I think you can express it more simply.

You don't need $$B$$ and $$\Psi$$ or $$\alpha$$ in your equations. These aren't necessary for the simple model you specify. You can ignore $$\delta$$ for this model, if you want to keep things simple and just standardize everything.

If you do this, then the implied correlation matrix is given by:

$$\Sigma(\Theta) = \Lambda \Lambda ^T$$

The sample correlation matrix is S, and you fit the model by minimizng $$F_{ML}$$:3

$$F_{ML} = log(det(S)) + tr(S \Sigma(\Theta^{-1}) - log(det(\Sigma(\Theta)) - p$$

Where det is the determinant, and tr is the trace of the matrix, and p is the number of parameters estimated.

This is the simplest case, other more complex cases extend this, but the basic idea is the same: Find the implied covariance matrix, given the parameters. Minimize F.

I hope this is what you were asking. :) If not, feel free to ask for clarification.

(I wrote a paper about this once: https://www.researchgate.net/publication/7151927_Confirmatory_factor_analysis_using_Microsoft_Excel)

• Oh, sorry, I realize you were using alpha differently to the way I extended. Ignore that part. – Jeremy Miles Nov 28 '19 at 19:05
• Thanks again Jeremy. I don't think it is possible to ignore the other matrices, since in this case Lambda*Lambda' (I don't know either how to add math to comments) results in a 3x3 identitiy matrix so there are no parameters that could be manipulated to try an minimize F_ML. But I think this is my fault for using an unintuitive example. Maybe my main question with regards to your answer is: what would you say are the parameters in Theta in your description (Sigma(Theta))? – hendogg87 Nov 29 '19 at 12:15
• I still think they must be alpha^2 and delta_t^2. And I know there is an explicit solution to each because I can work it out with least squares (the part I deleted from the original question). So my intuition is that there must be a way to solve either Sigma(theta) or F_ML to give me the variances of alpha and delta (the between- and within-person variance, respectively). – hendogg87 Nov 29 '19 at 12:26
• To avoid an extended discussion here, I created a chat room. chat.stackexchange.com/rooms/101673/… – Jeremy Miles Dec 1 '19 at 6:44