27
Latent Class Analysis is in fact an Finite Mixture Model (see here). The main difference between FMM and other clustering algorithms is that FMM's offer you a "model-based clustering" approach that derives clusters using a probabilistic model that describes distribution of your data. So instead of finding clusters with some arbitrary chosen distance measure, ...
19
An answer to Topic models and word co-occurrence methods covers the difference (skip-gram word2vec is compression of pointwise mutual information (PMI)).
So:
neither method is a generalization of another,
word2vec allows us to use vector geometry (like word analogy, e.g. $v_{king} - v_{man} + v_{woman} \approx v_{queen}$, I wrote an overview of word2vec)
...
18
This is an excellent question.
Probabilistic PCA (PPCA) is the following latent variable model
\begin{align}
\mathbf z &\sim \mathcal N(\mathbf 0, \mathbf I) \\
\mathbf x &\sim \mathcal N(\mathbf W \mathbf z + \boldsymbol \mu, \sigma^2 \mathbf I),
\end{align}
where $\mathbf x\in\mathbb R^p$ is one observation and $\mathbf z\in\mathbb R^q$ is a ...
answered Apr 22 '16 at 15:08
amoeba says Reinstate Monica
76.4k20 gold badges238 silver badges286 bronze badges
13
The two algorithms differ quite a bit in their purpose.
LDA is aimed mostly at describing documents and document collections by assigning topic distributions to them, which in turn have word distributions assigned, as you mention.
word2vec looks to embed words in a latent factor vector space, an idea originating from the distributed representations of ...
11
The root of the difficulty you are having lies in the sentence:
Then using the EM algorithm, we can maximize the second
log-likelihood.
As you have observed, you can't. Instead, what you maximize is the expected value of the second log likelihood (known as the "complete data log likelihood"), where the expected value is taken over the $z_i$.
This ...
11
Expectations are central to the EM algorithm. To start with, the likelihood associated with the data $(x_1,\ldots,x_n)$ is represented as an expectation
\begin{align*}
p(x_1,\ldots,x_n;\theta) &= \int_\mathfrak{{Z}^n} p(x_1,\ldots,x_n,\mathfrak{z}_1,\ldots,\mathfrak{z}_n;\theta)\,\text{d}\mathbf{\mathfrak{z}}\\
&=\int_\mathfrak{{Z}^n} p(x_1,\ldots,...
10
To choose an optimal number of latent factors in non-negative matrix factorization, use cross-validation.
As you wrote, the aim of NMF is to find low-dimensional $\mathbf W$ and $\mathbf H$ with all non-negative elements minimizing reconstruction error $\|\mathbf V-\mathbf W\mathbf H\|^2$. Imagine that we leave out one element of $\mathbf V$, e.g. $V_{ab}$, ...
answered Oct 29 '14 at 17:51
amoeba says Reinstate Monica
76.4k20 gold badges238 silver badges286 bronze badges
9
It would be possible to create a dataset in such a way that the residuals from the linear model are orthogonal to the quadratic and cubic terms and in that case adding the terms would not change the fit to the model. However the probability of getting such a dataset in the real world is close enough to 0 that you will not likely ever see this happen with a ...
9
Not necessarily.
Or, perhaps a better answer is, "it depends on what you mean by causality".
It is presumed, in the latent factor model, that the individual items measure the latent factor; that's really the point. So, on, e.g. the MMPI, the different questions are supposed to be measuring aspects of personality, and the purpose of factor analysis is to ...
answered Jun 11 '13 at 11:59
9
The covariance matrix of the data is always non-negative definite, there is no doubt about that. However, the model-implied covariance matrix may not be when some parameters take values outside their natural ranges. In turn, this may happen for a number of reasons.
Your 4-factor model may be misspecified, i.e., does not fit the data right.
Your model is OK,...
9
They're the same model.
It's useful to be able to define a latent variable as a composite outcome where that variable only has composite indicators.
If you don't have:
f1 =~ y1 + y2 + y3
You can't put:
f1 ~ x1 + x2 + x3
But you can have:
f1 <~ x1 + x2 + x3
7
(You asked whether there is a statistical reason: I doubt it, but I'm guessing about other reasons.) Would there be cries of "moving the goalpost"? Students usually like to know when taking a test just how much each item is worth. They might be justified in complaining upon seeing, for example, that some of their hard-worked answers didn't end up ...
7
See Tueller (2010), Tueller and Lubke (2010), and [Ruscio et al.'l book][3] for complete detail on what is summarized below. Taxometric procedures generally work by computing simple statistics on subset of sorted data. MAMBAC uses the mean, MAXCOV uses the covariance, and MAXEIG using the eigen value. Latent class analysis is a special case of the general ...
7
One common means of controlling for some other covariate would be via regression. Put the X and Y values into the response (DV), and a Y-group indicator (0 if in X, 1 if in Y) as a DV, along with your covariate (or some suitable proxy for it if the variable can't be measured directly) as another DV.
You regress on your covariate and the group-indicator. If ...
answered May 22 '15 at 1:59
Glen_b -Reinstate Monica
231k25 gold badges468 silver badges818 bronze badges
6
In the paper, and in general, (random) variables are everything which is drawn from a probability distribution. Latent (random) variables are the ones you don't directly observe ($y$ is observed, $\beta$ is not, but both are r.v). From a latent random variable you can get a posterior distribution, which is its probability distribution conditioned to the ...
6
PPCA was introduced in Tipping & Bishop, 1999, Probabilistic Principal Component Analysis. I would say that this paper itself is one of the best references: it is concise and clear.
Nevertheless, it might be difficult for a beginner. If so, you can try Bishop's textbook Pattern Recognition and Machine Learning, which is excellent and includes a thorough ...
answered Mar 22 '16 at 21:28
amoeba says Reinstate Monica
76.4k20 gold badges238 silver badges286 bronze badges
6
Probabilistic canonical correlation analysis (probabilistic CCA, PCCA) was introduced in Bach & Jordan, 2005, A Probabilistic Interpretation of
Canonical Correlation Analysis, several years after Tipping & Bishop presented their probabilistic principal component analysis (probabilistic PCA, PPCA).
Very briefly, it is based on the following ...
answered Apr 14 '16 at 10:48
amoeba says Reinstate Monica
76.4k20 gold badges238 silver badges286 bronze badges
6
Your question is unclear: when simulating $X\sim f(x)$, one can instead simulate$$(X,U)\sim\mathcal{U}(\{(x,u);\ 0<u<f(x)\})$$which has the joint density$$\mathbb{I}_{(0,f(x)}(u)\mathbb{I}_\mathcal{X}(x)$$and then use only the $X$ component in the simulation. For instance, here is a figure from our book depicting many realisations of such a Uniform ...
5
You may be thinking that it might be possible that the length of time somebody is in school may affect how uncertain we are about their education. I suspect that what you were told stems from a different way of thinking about this.
If the number of years in school is measured accurately, for example from reliable administrative records, then there should ...
5
So is the intuition correct that I shouldn't be concerned that the
overparameterized latent variables are not identifiable and aren't
fully converged when I take my posterior samples?
I think your intuition is correct: you shouldn't be concerned that the overparameterized latent variables are not identifiable and aren't
fully converged. In fact, the ...
5
The complete data likelihood should not involve G! It should simply be the likelihood of $\theta$ when the $X$'s are exponential. Note that the complete data likelihood as you have it written simplifies to an exponential likelihood since only one of the $G_{rj}$'s can be 1. Leaving the $G$'s in the complete data likelihood, however, messes you up later on. ...
5
The equation lists $\prod_{i\not=j} q_i dz_i$ as not a constant because it's a multiplicative factor to $q_i$ and is important from optimization perspective. Further down a formula (10.9) for $q_i$ update is derived and it heavily depends on these $\{q_j\}_{j\not=i}$.
From the other hand, $\int q_i \sum_{i\not=j} \ln q_j$ is left aside as $const$ because it'...
5
There is a lot of confusion in the question, confusion that could be reduced by looking at a textbook on the paper, or even the original 1977 paper by Dempster, Laird and Rubin.
Here is an excerpt of our book, Introducing Monte Carlo Methods with R, followed by my answer:
Assume that we observe $X_1, \ldots, X_n$, jointly distributed from $g({\mathbf x}|...
4
The OpenMx project can estimate growth mixture models, though you have to install the package from their website since it isn't on CRAN. They have examples in the user documentation (section 2.8) for how to set this up as well.
4
consider the two measurement model for Anxiety: in your first model you have given all three items equal weight, while in the second model you estimated the relative contribution of each of the items (in this case relative to item 3, so the loading for item 3 is 1). By fitting the relative contributions instead of constraining them all to be equal you are ...
4
I routinely use bnlearn in R on networks of that size and larger. Most likely your network is too highly connected to be solved by bnlearn: the cliques and conditional probability tables are too large.
I recommend using your domain knowledge to only impose an ordering on your variables and then learn the full structure and parameters from data.
See the ...
4
I don't have any good references for you, but yes, you can do this. The question will be what the results mean and that would depend on the specifics of the tests. If they have good alphas as single tests, that means the averaged correlations among items aren't too bad. That's an aspect of reliability. But what about validity? The fact that the items ...
answered Dec 4 '13 at 11:00
4
Based off @jsk's comments I will try to remedy my mistakes:
$\begin{align*}
L(\theta|X,G) &= \prod_{j=1}^n \theta e^{-\theta x_j}
\end{align*}$
$\begin{align*}
Q(\theta,\theta^i) &= n\log{\theta} - \theta\sum_{j=1}^n \text{E}\left[X_j|G,\theta^i\right]\\
&= n\log{\theta} - \theta\left(\dfrac{\sum_{j=1}^n g_{1j}}{1-e^{-\theta^i}}\right)\left(\...
4
You don't necessarily need to change $U$ and $V$ to be binary as well; in fact, doing so will probably be somewhat harmful. Remember that $\mathbb{E} R_{ij} = \sum_k U_{ik} V_{jk}$, so that if $U$ and $V$ have binary entries, for $\mathbb{E} R_{ij}$ to be 0 or 1, at most one corresponding pair of entries can be 1.
One thing you could do is just to use the ...
Only top voted, non community-wiki answers of a minimum length are eligible
