2
$\begingroup$

In a short paper titled Discriminative models, not discriminative training by Tom Minka, it says that the discriminative training might work better because it has two sets of independent parameters and hence is more flexible. Then I just wonder why we should model the $p(x_i|\theta')$ as stated in that paper? Or how we get it while training the discriminative models? I learned before that the $X$ is given and then it is unnecessary to encode a distribution over it, and in practice we often use only one set of parameters, like we train a logistic regression model and etc.

More specifically, if we don't need to care about the $P(X)$(as said in the paper "the best θ(in (7)) is the same as in (3)") that statement would not hold and what is the value of that paper? If we do how should we train the discriminative model, for instance logistic regression, using (7)? Any examples/references of discriminative models and the training according to (7)?

enter image description here

I thought it just means that while we are training the discriminative models, like logistic regression, we are actually training the joint distribution the same as the generative models?

Improve this question
$\endgroup$
3
  • $\begingroup$ I think $p(x_i|\theta)$ corresponds to the prior distribution over the classes. $\endgroup$
    – naive
    Commented Apr 28, 2019 at 10:30
  • $\begingroup$ @naive No the $x_i$ are the feature vectors $\endgroup$
    – TheCG
    Commented Apr 28, 2019 at 10:53
  • $\begingroup$ Sorry about that. I interpreted OP's question as asking ...I learned before that the X is given and then it is unnecessary to model it..... I was not specifically referring to $x_i$ but over $x$. And, as mentioned in the document linked -- $p(x|\theta) = \sum p(x, c|\theta)$ $\endgroup$
    – naive
    Commented Apr 28, 2019 at 11:02

2 Answers 2

Reset to default
2
$\begingroup$

You are right that discriminative models have two sets of parameters. You are also right that in practice only one set of parameters is used. This is not a contradiction. The paper is about having the right point of view about what you are doing. When you are doing logistic regression, you should have (7) in your mind and realize that you are throwing away one set of parameters. This act of throwing away is built into the logistic regression algorithm and will always happen when using (7). You may try to fit $\theta'$ but it won't matter since it is independent from $\theta$.

You are right to ask whether it is a good idea to throw away that extra set of parameters. In many situations, $\theta'$ has useful information. To make any use of $\theta'$, you must use a model different from (7). Some people think that (2) is the only alternative to (7), but the paper suggests a more general approach that includes (2) and (7) as special cases. This has proven successful in later work, such as:

  1. Principled Hybrids of Generative and Discriminative Models
  2. The linear "Bayes Point Machine" classifier built into the Azure Machine Learning toolkit, which uses $\theta$ and $\theta'$ under the hood.

None of that work would have been possible without the point of view given by this paper.

Improve this answer
$\endgroup$
1
$\begingroup$

As I understand, the point of the author is precisely to show that if you do not model $p(x_i|\theta)$ in discriminative training (Equation (3)) you are working with a different model than the one you would work with in the generative training.

He proposes a generalization of the base model (Equation (7)) which is then trained in both generative and discriminative contexts. In both cases $Q(D,\theta,\theta')$ is maximized, but what changes is the factorization of $Q(D,\theta,\theta')$ you use.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for the explanation. I am still confused that when we use discriminative models, like logistic regression, we don't need to care about the $p(x_i|θ)$. Could you please help further explain that a bit? Thanks. $\endgroup$
    – Lerner Zhang
    Commented Apr 28, 2019 at 12:39
  • $\begingroup$ @LernerZhang Indeed you don't model $p(x_i|\theta)$ in logistic regression. Look at these lecture notes. On the 5th page, $\mathcal{D}(x)$ is ignored in the maximization : starting from the maximization on the joint distribution we fall back on the problem solved in discriminative context. But since a term is simply neglected, the models upon which the maximization is done are not the same. I think this is the point of the article of your question. $\endgroup$
    – TheCG
    Commented Apr 28, 2019 at 13:06
  • $\begingroup$ I thought the author means that the discriminative training of a generative model is actually not for the same generative model as we do generative training. Am I right? $\endgroup$
    – Lerner Zhang
    Commented May 3, 2019 at 2:46
  • $\begingroup$ That's also what I understand from the paper! $\endgroup$
    – TheCG
    Commented May 3, 2019 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.