I know that recurrent neural network is used to generate text and to model the probability of $P(x_0,x_1,x_2,x_3)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)P(x_3|x_2,x_1,x_0)$ where $x_i$ is words/text.
If RNN can model probability, Can we calculate the probability of a text?
for example: It produces low probability for text with no meaning and high probability for normal text.
What about CNN, Can we generate probability for images(for example: low probability for fake images)?


1 Answer 1


Yes, in fact that's how most language models are trained -- you compute the (log) probability of given input text, and then backpropagate to maximize that probability.

There are some generative models with explicit densities: you can easily evaluate the probability/density the model assigns to any data point. There are some other generative models with only approximate densities (you can only get a lower bound on the probability of a point) -- or with implicit density (in which case finding or approximating the density is computationally intractable).

RNNs (and more broadly, autoregressive models) are in the explicit density family. There are many CNN based generative models of images (autoregressive, GAN, VAE, flows, etc), some of which are explicit, approximate, and implicit density each.

  • $\begingroup$ Thank you so much for your answer. can you please provide an example? for example: If I want to calculate the probability of a text, Should I build a RNN(seq2seq?) to output the same input and then after training, I take the product of the output probabilities?? $P(x_0,x_1,x_2)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)$ . Am I right? $\endgroup$
    – floyd
    Sep 10, 2019 at 5:06
  • $\begingroup$ check out tensorflow.org/tutorials/sequences/recurrent $\endgroup$
    – shimao
    Sep 10, 2019 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.