38 votes
Accepted

Latent Class Analysis vs. Cluster Analysis - differences in inferences?

Latent Class Analysis is in fact an Finite Mixture Model (see here). The main difference between FMM and other clustering algorithms is that FMM's offer you a "model-based clustering" approach that ...
user avatar
  • 115k
29 votes
Accepted

What is principal subspace in probabilistic PCA?

This is an excellent question. Probabilistic PCA (PPCA) is the following latent variable model \begin{align} \mathbf z &\sim \mathcal N(\mathbf 0, \mathbf I) \\ \mathbf x &\sim \mathcal N(\...
user avatar
  • 95.1k
20 votes
Accepted

LDA vs word2vec

An answer to Topic models and word co-occurrence methods covers the difference (skip-gram word2vec is compression of pointwise mutual information (PMI)). So: neither method is a generalization of ...
user avatar
  • 5,616
15 votes
Accepted

How to choose an optimal number of latent factors in non-negative matrix factorization?

To choose an optimal number of latent factors in non-negative matrix factorization, use cross-validation. As you wrote, the aim of NMF is to find low-dimensional $\mathbf W$ and $\mathbf H$ with all ...
user avatar
  • 95.1k
15 votes

LDA vs word2vec

The two algorithms differ quite a bit in their purpose. LDA is aimed mostly at describing documents and document collections by assigning topic distributions to them, which in turn have word ...
user avatar
  • 2,542
14 votes
Accepted

When a CFA model has a "covariance matrix was not positive definite" problem, is it due to the dataset or the model?

The covariance matrix of the data is always non-negative definite, there is no doubt about that. However, the model-implied covariance matrix may not be when some parameters take values outside their ...
user avatar
  • 29.4k
12 votes
Accepted

Why is there a E in the name EM algorithm?

Expectations are central to the EM algorithm. To start with, the likelihood associated with the data $(x_1,\ldots,x_n)$ is represented as an expectation \begin{align*} p(x_1,\ldots,x_n;\theta) &= \...
user avatar
  • 92.7k
9 votes
Accepted

Parameters vs latent variables

In the paper, and in general, (random) variables are everything which is drawn from a probability distribution. Latent (random) variables are the ones you don't directly observe ($y$ is observed, $\...
user avatar
  • 2,696
9 votes
Accepted

What's the difference between a MIMIC factor and a composite with indicators (SEM)?

They're the same model. It's useful to be able to define a latent variable as a composite outcome where that variable only has composite indicators. If you don't have: ...
user avatar
  • 14.8k
9 votes

Is it true that the word prior should be used only with latent random variables?

The term prior (as well as posterior) is usually reserved for distributions defined in a Bayesian framework on objects that are not considered as random variables by other inferential approaches, ...
user avatar
  • 92.7k
8 votes
Accepted

Beginner references to understand probabilistic principal component analysis (PPCA)

PPCA was introduced in Tipping & Bishop, 1999, Probabilistic Principal Component Analysis. I would say that this paper itself is one of the best references: it is concise and clear. Nevertheless, ...
user avatar
  • 95.1k
7 votes

Student's t-test with a covariate?

One common means of controlling for some other covariate would be via regression. Put the X and Y values into the response (DV), and a Y-group indicator (0 if in X, 1 if in Y) as a DV, along with your ...
user avatar
  • 261k
7 votes
Accepted

The fundamental theorem of simulation

Your question is unclear: when simulating $X\sim f(x)$, one can instead simulate$$(X,U)\sim\mathcal{U}(\{(x,u);\ 0<u<f(x)\})$$which has the joint density$$\mathbb{I}_{(0,f(x)}(u)\mathbb{I}_\...
user avatar
  • 92.7k
6 votes

How to choose an optimal number of latent factors in non-negative matrix factorization?

To my knowledge, there are two good criteria: 1) the cophenetic correlation coefficient and 2) comparing the residual sum of squares against randomized data for a set of ranks (maybe there is a name ...
user avatar
6 votes
Accepted

Probabilistic models for partial least squares, reduced rank regression, and canonical correlation analysis?

Probabilistic canonical correlation analysis (probabilistic CCA, PCCA) was introduced in Bach & Jordan, 2005, A Probabilistic Interpretation of Canonical Correlation Analysis, several years after ...
user avatar
  • 95.1k
6 votes
Accepted

ELBO maximization with SGD

I think you confuse the purpose of the two methods. Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler ...
user avatar
6 votes

Is it true that the word prior should be used only with latent random variables?

Before answering your question, let's first explain some basic Bayesian mindset. In Bayesian statistics, everything is a random variable, the only difference between these random variables is whether ...
user avatar
6 votes
Accepted

Interpretating SEM estimates for 2 different datasets

Small things first: I'm not sure what you mean by the high dimension of the dataset mattering. OLS is ML, so the fact that you used ML instead of OLS doesn't matter. The structural part of your model ...
user avatar
  • 14.8k
5 votes
Accepted

Mean field variational inference

The equation lists $\prod_{i\not=j} q_i dz_i$ as not a constant because it's a multiplicative factor to $q_i$ and is important from optimization perspective. Further down a formula (10.9) for $q_i$ ...
user avatar
5 votes
Accepted

EM algorithm Practice Problem

The complete data likelihood should not involve G! It should simply be the likelihood of $\theta$ when the $X$'s are exponential. Note that the complete data likelihood as you have it written ...
user avatar
  • 2,871
5 votes
Accepted

Why does probabilistic PCA use Gaussian prior over latent variables?

Probabilistic PCA Probabilistic PCA is a Gaussian latent variable model of the following form. Observations $\mathbf x \in \mathbb R^D$ consist of $D$ variables, latent variables $\mathbf z \in \...
user avatar
  • 95.1k
5 votes
Accepted

How and When to Use Marginalization in Stan

Stan only samples from continuous parameter spaces, so for something like a finite mixture model, it is necessary to do marginalization to use Stan. On the other hand, if you have a hierarchical model ...
user avatar
  • 1,396
5 votes
Accepted

Question about the latent variable in EM algorithm

There is a lot of confusion in the question, confusion that could be reduced by looking at a textbook on the paper, or even the original 1977 paper by Dempster, Laird and Rubin. Here is an excerpt ...
user avatar
  • 92.7k
5 votes
Accepted

Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator

You could express the complete log-likelihood as $$\ell(\theta) = -n\log\sigma_1-\frac{1}{2}\sum_{i=1}^n \sigma_1^{-2}(z_i-\mu_1)^2 -n\log\sigma^2-\frac{1}{2}\sum_{i=1}^n \sigma^{-2}(y_i-\beta_1-\...
user avatar
  • 92.7k
5 votes

Latent Profile Analysis and Statistical Methods in Psychology

Latent Profile Analysis (LPA) is a term typically used for a model which identifies latent sub-populations within a population based on a certain set of categorical variables. In your case, you have ...
user avatar
  • 49.8k
5 votes
Accepted

Finding a Common Thread in Disparate Indicators

Have you considered the bifactor measurement model? From what I can glean from your question, it looks as though you are interested in a single social norms dimension, with an item bank that could ...
user avatar
4 votes
Accepted

Binary version of Probabilistic Matrix Factorization in pymc?

You don't necessarily need to change $U$ and $V$ to be binary as well; in fact, doing so will probably be somewhat harmful. Remember that $\mathbb{E} R_{ij} = \sum_k U_{ik} V_{jk}$, so that if $U$ and ...
user avatar
  • 22.3k
4 votes

Latent Class Analysis vs. Cluster Analysis - differences in inferences?

A latent class model (or latent profile, or more generally, a finite mixture model) can be thought of as a probablistic model for clustering (or unsupervised classification). The goal is generally the ...
user avatar
  • 3,683
4 votes

Latent Class Analysis vs. Cluster Analysis - differences in inferences?

The difference is Latent Class Analysis would use hidden data (which is usually patterns of association in the features) to determine probabilities for features in the class. Then inferences can be ...
user avatar
  • 1,018
4 votes

EM algorithm Practice Problem

Based off @jsk's comments I will try to remedy my mistakes: $\begin{align*} L(\theta|X,G) &= \prod_{j=1}^n \theta e^{-\theta x_j} \end{align*}$ $\begin{align*} Q(\theta,\theta^i) &= n\log{\...
user avatar
  • 8,672

Only top scored, non community-wiki answers of a minimum length are eligible