Generate images from CNN model built with transfer learning

I have trained an image classifier (for my own face) via the transfer learning method, in two ways (in order to compare efficiencies later on):

1- I created a docker image, built on top of tensorflow's mobilenet network, and trained it with my own dataset. training was succesful and generated two files:

  /retrained_graph.pb
retrained_labels.txt


2- using kerasapi, training my dataset on top of inception network, which generated the single file:

   /inceptionv3-ft.model


if I did everything right, from both methods the top network layer was removed and repurposed with my data.

my goal here is to generate googledream-like, images (or any other method for that matter) from (either one of) these newly created models.

An example of what I want to achive (model trained for the face):

that is, knowing the class (y) and knowing my custom parameters (W), solve the model for X, and generate the image, instead of classifying it:

X . W = y

could someone please point me in the right direction and tell me how I could use these models for that purpose?

The intuition behind "neural network dreams" is to "turn the neural network upside down" and instead of using it to compute the probability that the input image belongs to a certain class, use it to modify the input image, for fixed network weights, until the probability of it belonging to a given class is maximized. The first step is to define a loss function, which in this case could be the input of the output neuron corresponding to class Cat. We could also use the output of that neuron, i.e., the probability that the image is Cat, but empirically it has been found that the first choice results in better-looking images. Then, using automatic differentiation, you compute the gradient of the loss function with respect to the input features, i.e., to the pixels of the input image, not with respect to the weights of the neural networks as you do in backpropagation. Here the neural network weights are fixed.

For example, you could start with a random noise image, and compute the gradient of the activation of class Cat with respect to the input features. Now you add this gradient (multiplied by a suitable step size) to the input image, so that activation is increased. We call this process gradient ascent, because we're trying to maximize the loss (activation) instead than minimizing it as we do with (stochastic) gradient descent. This approach by itself doesn't work very well: it results in images which are basically high-frequency, apparently random patterns, which look nothing like a cat (see for example here). However, if we add some statistical constraints which are usually satisfied by natural images, such as the fact that neighboring pixels are strongly correlated, the results are better.

However, this still doesn't result in very "dreamy" images. Deep Dream uses a different approach: we don't select any class, and we don't start from a random noise image. Instead, we choose one of the neural network layers, and we give the neural network an input image. This time our loss is the mean of the layer output (before applying the ReLu nonlinearity, usually, because otherwise gradients for features with negative initial activations will be zero). Correspondingly, the gradient is the gradient of this mean activation with respect to the input features (the pixel values of the input images). Now gradient ascent will enhances whatever features increase the most the mean layer activation the most. In other words, it enhances the features detected by that layer. This results in "applying a theme" to your input image, where the theme depends from the selected layer. Each layer of the network deals with features at a different level of abstraction, so the complexity of features we generate depends on which layer we choose to enhance. For example, lower layers tend to produce strokes or simple ornament-like patterns, because those layers are sensitive to basic features such as edges and their orientations.

The algorithm is usually applied in an iterative fashion, first downscaling the input image to a minimum size (image shape, i.e., number of pixels in the vertical & horizontal direction), then applying gradient ascent, then upscaling to a successive size, until we get back to the original image size. The successive scales are called octaves. The DeepDream algorithm was originally implemented in Caffe, so it's useless for most practitioners today. Luckily, there are modern implementations in Keras and Tensorflow:

https://github.com/keras-team/keras/blob/master/examples/deep_dream.py (classic Keras implementation: only works with Inception v3, you need to modify the code manually to get it to work with other networks)

https://github.com/danielvarga/keras-deep-dream (slightly more general Keras implementation: the neural network model and the layer name can be passed to the script as CL arguments, without having to manually modify the code)

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb (Tensorflow implementation, it uses the Inception architecture but I'm not sure which version)

• brilliant explanation. but I'd like to know if my trained model above (on top of inception) can be chosen as the fixed neural network layer for that purpose, for a given input. dreamy qualities are not mandatory here. my main goal is to make my retrained class 'appear' on a new input. could you please expand your (great) answer on that? – 8-Bit Borges Dec 28 '17 at 19:06
• @data_garden you can use other models with Deep Dream. However you need to modify the code manually, importing your architecture instead than a pre-trained one. I won't modify the answer because this is not a programming site - here the discussion on the "theory" of the algorithm is on-topic, not its implementation in a Python script. Anyway, if you have some Keras/Tensorflow skills you should be able to modify the code, now that you've understood the underlying principle. – DeltaIV Dec 28 '17 at 19:34
• @data_garden wait - are you maybe interested in a minimal perturbation $\epsilon$ to an input image, originally classified as class $K$ with high probability, so that now the perturbed image is classified with high probability as class $M$ with $K\neq M$? If so, you need to edit your question accordingly - this is a completely different problem, an adversarial attack. – DeltaIV Dec 28 '17 at 19:43
• I haven't heard of this method you mention. My actual case: I have trained my face (K?) making use of the inception network. and now I want to make a (minimal perturbation ?) on an input image of clouds, 'printing' my face on it. makes sense? if so, I'll edit the question. google dream was just an example of a method solving for X. – 8-Bit Borges Dec 28 '17 at 19:56
• Oh wow, that's not even an adversarial attack... that sounds more like style transfer. You want the same effect than the Prism app generates. Right? That might be doable..but it's a very different problem. It could be also approximated through Deep Dream. Let's talk in chat tomorrow and see what your exact requirements are, then I'll help you ask another more accurate answer. – DeltaIV Dec 28 '17 at 20:07

DeepDream has no network architecture. It is just slicing through the different layers of a deep network to extract out different visualization possible.

An implementation is shown below:

https://github.com/nate-parrott/juypter-notebooks/blob/master/dream.ipynb

Essentially different layers in network architecture (which ever it is, can be different types of convolution network etc) is extracted and rendered.