Why are the additional set of parameters in discriminative models necessary(in Minka's 2005 paper)? In a short paper titled Discriminative models, not discriminative training by Tom Minka, it says that the discriminative training might work better because it has two sets of independent parameters and hence is more flexible. Then I just wonder why we should model the $p(x_i|\theta')$ as stated in that paper? Or how we get it while training the discriminative models? I learned before that the $X$ is given and then it is unnecessary to encode a distribution over it, and in practice we often use only one set of parameters, like we train a logistic regression model and etc. 
More specifically, if we don't need to care about the $P(X)$(as said in the paper "the best θ(in (7)) is the same as in (3)") that statement would not hold and what is the value of that paper? If we do how should we train the discriminative model, for instance logistic regression, using (7)? Any examples/references of discriminative models and the training according to (7)? 

I thought it just means that while we are training the discriminative models, like logistic regression, we are actually training the joint distribution the same as the generative models? 
 A: You are right that discriminative models have two sets of parameters.  You are also right that in practice only one set of parameters is used.  This is not a contradiction.  The paper is about having the right point of view about what you are doing.  When you are doing logistic regression, you should have (7) in your mind and realize that you are throwing away one set of parameters.  This act of throwing away is built into the logistic regression algorithm and will always happen when using (7).  You may try to fit $\theta'$ but it won't matter since it is independent from $\theta$.
You are right to ask whether it is a good idea to throw away that extra set of parameters.  In many situations, $\theta'$ has useful information.  To make any use of $\theta'$, you must use a model different from (7).  Some people think that (2) is the only alternative to (7), but the paper suggests a more general approach that includes (2) and (7) as special cases.  This has proven successful in later work, such as:


*

*Principled Hybrids of Generative and Discriminative Models

*The linear "Bayes Point Machine" classifier built into the Azure Machine Learning toolkit, which uses $\theta$ and $\theta'$ under the hood.


None of that work would have been possible without the point of view given by this paper.
A: As I understand, the point of the author is precisely to show that if you do not model $p(x_i|\theta)$  in discriminative training (Equation (3)) you are working with a different model than the one you would work with in the generative training.
He proposes a generalization of the base model (Equation (7)) which is then trained in both generative and discriminative contexts. In both cases $Q(D,\theta,\theta')$ is maximized, but what changes is the factorization of $Q(D,\theta,\theta')$ you use.
