Discriminative vs. Generative Models

Question

This has be asked before, but I still have not grasped it completely. I know that generative models model the feature distribution and that this includes modelling the P(x|y) and P(y), which are not required if we are trying to classify (find P(y|x)).

Question: Many text books say that it is easier to include features in discriminative models which is rarely explained. Also they mention that discriminative models allow overlapping features (features that are interdependent). Could anybody explain what this means and why is it true, or guide me to the place to read? I see that it is possible to include features in generative models as well, and I can't see why it should be easier or more efficient in discriminative models.

Answer 1

OK this is only a partial answer based on what I understand so far:

1. It is easier to make the dataset separable (either linearly or not) by applying a feature extraction step (feature transformation or a preprocessing step as it is called sometimes) before classification than making the dataset's class-conditional probability distributions follow an assumption (Gaussian or Uniform ...). Why? not entirely sure but it seems to be that assuming a probability distribution is a stronger assumption than the linear separability in the case of logistic vs Bayesian. So, amusing that after the feature extraction these assumption is going to hold is even more ambitious than achieving linear separability by going to another possibly high dimensional space (like in Support Vector Machines).
2. In the literature, feature functions are included as part of the logistic regression model sometimes (and sometimes not). But never for the Generative counterpart. So theoretical convenience is my answer here but I am not convinced deep inside that this matters.

Happy to here some thoughts on this