What are the best resources on image synthesis?

Question

What are some good resources to learn about image synthesis? What are some of the key concepts or architectures to study?

I understand image synthesis as generating new images with ML techniques.

Answer 1

This is a book on Generative Models:

Tomczak, Jakub M. "Deep Generative Modeling". 2022.

To help readers to choose which models to study I summarize the main classes of Generative Models and provide a brief description of each, which are the resources I studied for an introductory course to Generative Models

AutoRegressive Models

The image is modeled as a sequence of pixel values. The image distribution $p(x)$ is factorized into a product of conditional distributions and the generation is an autoregressive prediction of the next pixel values based on the previous ones.

Relevant papers

Van Den Oord, Aäron, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel recurrent neural networks." International conference on machine learning. PMLR, 2016.

Chen, Xi, et al. "Pixelsnail: An improved autoregressive generative model." International Conference on Machine Learning. PMLR, 2018.

Menick, Jacob, and Nal Kalchbrenner. "Generating high fidelity images with subscale pixel networks and multidimensional upscaling." ICLR (2019)

Pros

• Simple
• Exact likelihood, allow sampling from the distribution

Cons

• Does not have a learned representation
• Inference is slow, because of the sequential nature

Normalizing Flows

Parameterizes $p(x)$ as an invertible deterministic transformation from a base density, such as a standard Gaussian

Relevant papers

Dinh, Laurent, David Krueger, and Yoshua Bengio. ”NICE: Non-linear independent components estimation." ICLR 2015 workshop

Dinh, Laurent, Jascha Sohl-Dickstein, and Samy Bengio. "Density estimation using Real NVP." ICLR 2017

Kingma, Durk P., and Prafulla Dhariwal. "Glow: Generative flow with invertible 1x1 convolutions." Advances in neural information processing systems 31 (2018)

Papamakarios, George, et al. "Normalizing Flows for Probabilistic Modeling and Inference." J. Mach. Learn. Res. 22.57 (2021): 1-64.

Kobyzev, Ivan, Simon JD Prince, and Marcus A. Brubaker. "Normalizing flows: An introduction and review of current methods." IEEE Transactions on pattern analysis and machine intelligence 43.11 (2020): 3964-3979.

Pros

• A very flexible and elegant formulation with exact likelihood
• Allows fast sempling
• Provides a latent representation

Cons

• Limited freedom in the choice of the architecture, for theoretical (invertibility) and computational constraints
• Latent space needs to have the same dimensionality as the output for invertibility

Latent Variable Models

Use an encoder network to map inputs onto a low-dimensional latent space and a decoder network to generate images from a latent code

Relevant papers

Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

Doersch, Carl. "Tutorial on variational autoencoders." arXiv preprint arXiv:1606.05908 (2016).

Kingma, Diederik P., and Max Welling. "An introduction to variational autoencoders." Foundations and Trends® in Machine Learning 12.4 (2019): 307-392.

Higgins et al, $\beta$-vae: Learning Basic Visual Concepts with a Constrained Variational Framework, ICLR 2017

Van Den Oord et al "Neural discrete representation learning." Neurips (2017).

Pros

• No architecture constraint
• Efficient to learn, flexible
• Allows sampling

Cons

• Does not have exact likelihood
• Blurry samples if wrongly tuned and without perceptual components in loss

Generative Adversarial Networks

Train two networks, a generator which produces synthetic data and a discriminator which distinguishes between synthetic and real data

Relevant papers

Goodfellow, Ian et al. Generative Adversarial Networks, NIPS’14

Goodfellow, Ian. Tutorial: Generative Adversarial Networks, NIPS’16

M .Mirza and S. Osindero, "Conditional generative adversarial nets." arXiv preprint arXiv:1411.1784 (2014)

Salimans, Tim, et al. "Improved techniques for training GANs." Advances in neural information processing systems 29 (2016).

Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.

Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017

Jolicoeur-Martineau, Alexia. "The relativistic discriminator: a key element missing from standard GAN." arXiv preprint arXiv:1807.00734 ICLR (2018).

W. Fedus, et al., Many Paths To Equilibrium: Gans Do Not Need To Decrease A Divergence At Every Step, ICLR’18

Schmidhuber, Jürgen. "Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization (1991)." Neural Networks 127 (2020): 58-66

Pros

• Sharp, higher resolution outputs compared to the previous classes
• Full freedom in architecture
• Fast sampling
• Latent representation

Cons

• No likelihood
• Training instability

Diffusion Models

Learn a denoising process which transforms random noise into an image

Relevant Papers

Sohl-Dickstein, Jascha, et al. "Deep unsupervised learning using nonequilibrium thermodynamics." International Conference on Machine Learning. PMLR, 2015.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems 33 (2020): 6840-6851.

Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." International Conference on Machine Learning. PMLR, 2021.

Ho, Jonathan, et al. "Cascaded Diffusion Models for High Fidelity Image Generation." J. Mach. Learn. Res. 23 (2022): 47-1.

Song, Yang, et al. "Consistency models." arXiv preprint arXiv:2303.01469 (2023).

Pros

• Stable training
• Great tractability/flexibility trade-off
• High quality samples

Cons

• Lower likelihood value (even with higher quality results)
• Slow to sample from (Consistency models is an emergent class which aims to alleviate that)

Text-to-Image generation

Finally, the section where all the hype is, with the previous model classes applied to text-to-image generation

Zhang, Han, et al. "Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.

Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Ramesh, Aditya, et al. "Hierarchical text-conditional image generation with clip latents." arXiv preprint arXiv:2204.06125 (2022).

Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in Neural Information Processing Systems 35 (2022): 36479-36494.

Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Answer 2

Not sure this is what you are looking for but, my preferred resource is the book by Rafael C. Gonzalez and Richard E. Woods titled Digital Image Processing, Pearson, and now in its fourth edition. It's a quite comprehensive text on image processing, it covers: spatial filtering, filtering in the frequency domain, image restoration, colour image processing, wavelets, etc. The book has twelve chapters and the last chapter covers ML methods for image classification, e.g. NNets and Deep Learning.