38
votes
Accepted
Latent Class Analysis vs. Cluster Analysis - differences in inferences?
Latent Class Analysis is in fact an Finite Mixture Model (see here). The main difference between FMM and other clustering algorithms is that FMM's offer you a "model-based clustering" approach that ...
Tim♦
- 115k
29
votes
Accepted
What is principal subspace in probabilistic PCA?
This is an excellent question.
Probabilistic PCA (PPCA) is the following latent variable model
\begin{align}
\mathbf z &\sim \mathcal N(\mathbf 0, \mathbf I) \\
\mathbf x &\sim \mathcal N(\...
- 95.1k
20
votes
Accepted
LDA vs word2vec
An answer to Topic models and word co-occurrence methods covers the difference (skip-gram word2vec is compression of pointwise mutual information (PMI)).
So:
neither method is a generalization of ...
- 5,616
15
votes
Accepted
How to choose an optimal number of latent factors in non-negative matrix factorization?
To choose an optimal number of latent factors in non-negative matrix factorization, use cross-validation.
As you wrote, the aim of NMF is to find low-dimensional $\mathbf W$ and $\mathbf H$ with all ...
- 95.1k
15
votes
LDA vs word2vec
The two algorithms differ quite a bit in their purpose.
LDA is aimed mostly at describing documents and document collections by assigning topic distributions to them, which in turn have word ...
- 2,542
14
votes
Accepted
When a CFA model has a "covariance matrix was not positive definite" problem, is it due to the dataset or the model?
The covariance matrix of the data is always non-negative definite, there is no doubt about that. However, the model-implied covariance matrix may not be when some parameters take values outside their ...
- 29.4k
12
votes
Accepted
Why is there a E in the name EM algorithm?
Expectations are central to the EM algorithm. To start with, the likelihood associated with the data $(x_1,\ldots,x_n)$ is represented as an expectation
\begin{align*}
p(x_1,\ldots,x_n;\theta) &= \...
- 92.7k
9
votes
Accepted
Parameters vs latent variables
In the paper, and in general, (random) variables are everything which is drawn from a probability distribution. Latent (random) variables are the ones you don't directly observe ($y$ is observed, $\...
- 2,696
9
votes
Accepted
What's the difference between a MIMIC factor and a composite with indicators (SEM)?
They're the same model.
It's useful to be able to define a latent variable as a composite outcome where that variable only has composite indicators.
If you don't have:
...
- 14.8k
9
votes
Is it true that the word prior should be used only with latent random variables?
The term prior (as well as posterior) is usually reserved for distributions defined in a Bayesian framework on objects that are not considered as random variables by other inferential approaches, ...
- 92.7k
8
votes
Accepted
Beginner references to understand probabilistic principal component analysis (PPCA)
PPCA was introduced in Tipping & Bishop, 1999, Probabilistic Principal Component Analysis. I would say that this paper itself is one of the best references: it is concise and clear.
Nevertheless, ...
- 95.1k
7
votes
Student's t-test with a covariate?
One common means of controlling for some other covariate would be via regression. Put the X and Y values into the response (DV), and a Y-group indicator (0 if in X, 1 if in Y) as a DV, along with your ...
- 261k
7
votes
Accepted
The fundamental theorem of simulation
Your question is unclear: when simulating $X\sim f(x)$, one can instead simulate$$(X,U)\sim\mathcal{U}(\{(x,u);\ 0<u<f(x)\})$$which has the joint density$$\mathbb{I}_{(0,f(x)}(u)\mathbb{I}_\...
- 92.7k
6
votes
How to choose an optimal number of latent factors in non-negative matrix factorization?
To my knowledge, there are two good criteria: 1) the cophenetic correlation coefficient and 2) comparing the residual sum of squares against randomized data for a set of ranks (maybe there is a name ...
6
votes
Accepted
Probabilistic models for partial least squares, reduced rank regression, and canonical correlation analysis?
Probabilistic canonical correlation analysis (probabilistic CCA, PCCA) was introduced in Bach & Jordan, 2005, A Probabilistic Interpretation of
Canonical Correlation Analysis, several years after ...
- 95.1k
6
votes
Accepted
ELBO maximization with SGD
I think you confuse the purpose of the two methods.
Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler ...
- 1,196
6
votes
Is it true that the word prior should be used only with latent random variables?
Before answering your question, let's first explain some basic Bayesian mindset.
In Bayesian statistics, everything is a random variable, the only difference between these random variables is whether ...
- 663
6
votes
Accepted
Interpretating SEM estimates for 2 different datasets
Small things first:
I'm not sure what you mean by the high dimension of the dataset mattering.
OLS is ML, so the fact that you used ML instead of OLS doesn't matter.
The structural part of your model ...
- 14.8k
5
votes
Accepted
Mean field variational inference
The equation lists $\prod_{i\not=j} q_i dz_i$ as not a constant because it's a multiplicative factor to $q_i$ and is important from optimization perspective. Further down a formula (10.9) for $q_i$ ...
- 2,601
5
votes
Accepted
EM algorithm Practice Problem
The complete data likelihood should not involve G! It should simply be the likelihood of $\theta$ when the $X$'s are exponential. Note that the complete data likelihood as you have it written ...
- 2,871
5
votes
Accepted
Why does probabilistic PCA use Gaussian prior over latent variables?
Probabilistic PCA
Probabilistic PCA is a Gaussian latent variable model of the following form. Observations $\mathbf x \in \mathbb R^D$ consist of $D$ variables, latent variables $\mathbf z \in \...
- 95.1k
5
votes
Accepted
How and When to Use Marginalization in Stan
Stan only samples from continuous parameter spaces, so for something like a finite mixture model, it is necessary to do marginalization to use Stan. On the other hand, if you have a hierarchical model ...
- 1,396
5
votes
Accepted
Question about the latent variable in EM algorithm
There is a lot of confusion in the question, confusion that could be reduced by looking at a textbook on the paper, or even the original 1977 paper by Dempster, Laird and Rubin.
Here is an excerpt ...
- 92.7k
5
votes
Accepted
Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator
You could express the complete log-likelihood as
$$\ell(\theta) = -n\log\sigma_1-\frac{1}{2}\sum_{i=1}^n \sigma_1^{-2}(z_i-\mu_1)^2
-n\log\sigma^2-\frac{1}{2}\sum_{i=1}^n \sigma^{-2}(y_i-\beta_1-\...
- 92.7k
5
votes
Latent Profile Analysis and Statistical Methods in Psychology
Latent Profile Analysis (LPA) is a term typically used for a model which identifies latent sub-populations within a population based on a certain set of categorical variables. In your case, you have ...
- 49.8k
5
votes
Accepted
Finding a Common Thread in Disparate Indicators
Have you considered the bifactor measurement model? From what I can glean from your question, it looks as though you are interested in a single social norms dimension, with an item bank that could ...
- 420
4
votes
Accepted
Binary version of Probabilistic Matrix Factorization in pymc?
You don't necessarily need to change $U$ and $V$ to be binary as well; in fact, doing so will probably be somewhat harmful. Remember that $\mathbb{E} R_{ij} = \sum_k U_{ik} V_{jk}$, so that if $U$ and ...
- 22.3k
4
votes
Latent Class Analysis vs. Cluster Analysis - differences in inferences?
A latent class model (or latent profile, or more generally, a finite mixture model) can be thought of as a probablistic model for clustering (or unsupervised classification). The goal is generally the ...
- 3,683
4
votes
Latent Class Analysis vs. Cluster Analysis - differences in inferences?
The difference is Latent Class Analysis would use hidden data (which is usually patterns of association in the features) to determine probabilities for features in the class. Then inferences can be ...
- 1,018
4
votes
EM algorithm Practice Problem
Based off @jsk's comments I will try to remedy my mistakes:
$\begin{align*}
L(\theta|X,G) &= \prod_{j=1}^n \theta e^{-\theta x_j}
\end{align*}$
$\begin{align*}
Q(\theta,\theta^i) &= n\log{\...
- 8,672
Only top scored, non community-wiki answers of a minimum length are eligible