13
$\begingroup$

In this popular question, high upvoted answer makes MLE and Baum Welch separate in HMM fitting.

For training problem we can use the following 3 algorithms: MLE (maximum likelihood estimation), Viterbi training(DO NOT confuse with Viterbi decoding), Baum Welch = forward-backward algorithm

BUT in Wikipedia, it says

The Baum–Welch algorithm uses the well known EM algorithm to find the maximum likelihood estimate of the parameters

So, what's the relationship between MLE and Baum–Welch algorithm?

My attempt: The objective for Baum–Welch algorithm is maximize likelihood, but it uses a specialized algorithm (EM) to solve the optimization. We still can maximize likelihood by using other methods such as gradient decent. This is why the answer make two algorithm separate.

Am I right and can anyone help me to clarify?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ In the scope of HMM the MLE is used in a supervised scenario, and the Baum-Welch in an unsupervised, scenario. $\endgroup$
    – David Batista
    Commented Oct 23, 2017 at 18:40

3 Answers 3

Reset to default
4
$\begingroup$

Refer to one of the answers (by Masterfool) from the question link you provided,

Morat's answer is false on one point: Baum-Welch is an Expectation-Maximization algorithm, used to train an HMM's parameters. It uses the forward-backward algorithm during each iteration. The forward-backward algorithm really is just a combination of the forward and backward algorithms: one forward pass, one backward pass.

And I agree with PierreE's answer here, Baum–Welch algorithm is used to solve maximum likelihood in HHM. If the states are known (supervised, labeled sequence), then other method maximizing MLE is used (maybe like, simply count the frequency of each emission and transition observed in the training data, see the slides provided by Franck Dernoncourt).

In the setting of MLE for HMM, I don't think you can just use gradient descent, since the likelihood (or, log-likelihood) doesn't have a closed-form solution and must be solved iteratively, same as the case in mixture models so then we turn to EM. (See more details in Bishop, Pattern Recognition book, chapter 13.2.1 Pg614)

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I’m late to the party: you can use gradient descent to optimize the likelihood. You use the forward algorithm to compute the joint probability (or the marginal probability, if you don’t know the states during training). The gradient with respect to the parameters is well defined. (See Salakhutdinov, Roweis, and Ghahramani, 2003 at ICML.) $\endgroup$
    – Arya McCarthy
    Commented Apr 4, 2021 at 0:49
1
$\begingroup$

This question has been here for a few months but this answer might help new readers, as a complement to David Batista's comment.

The Baulm-Welch algorithm (BM) is an expectation maximization algorithm to solve maximum likelihood estimation (MLE) in order to train your HMM when the states are unknown/hidden (unsupervised training).

But if you know the states, you can use an MLE method (which will not be the BM) to fit your model to the pair data/states in a supervised fashion.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is also what I gather. I've heard MLE referred to as "counting" colloquially, because that's all it does (estimate probabilities as a ratio of two counts), but this requires being able to count the hidden states of your HMM in the data. EM algorithms assume that there are latent variables of which you have no knowledge, and you try to estimate them from just their observable effect. In short, for PoS tagging: MLE needs PoS tags in the dataset, Baum-Welch doesn't. $\endgroup$
    – Mew
    Commented Jan 18, 2023 at 15:55
0
$\begingroup$

So, what's the relationship between MLE and Baum–Welch algorithm?

Expectation maximization(EM) algorithm is more general and Baum-Welch algorithm is simply an instantiation of it, and EM is an iterative algorithm for maximum likelihood(ML). Then Baum-Welch algorithm is also an iterative algorithm for maximum likelihood.

There normally are three optimization algorithms for maximum likelihood estimation(a frequentist approach): 1) gradient descent; 2) Markov Chain Monte Carlo; 3) expectation maximization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.