2
$\begingroup$

I guess this is kind of an offbeat question. This is a situation where I understand the "how" and not the "why". Since this may not be well known, I'll provide an explanation first.

The question is concerning the calculation of the style cost when doing neural style transfer between two images (to combine one image which provides content with another that provides style, like making a photo you snapped of your dog look like a Picasso painting). It's a machine learning technique and pretty neat.

To generate this image, you need a content cost (how similar your resulting image is to your "content" image, in this case the picture of your dog) and a style cost (how similar your resulting image is to the Picasso painting). I won't describe the content cost here, only the style cost. Your overall cost is just the sum of the two, though.

To get the style cost, you are supposed to run both images through a pretrained neural network and look at the activations of a particular layer. You can choose the layer you want, I won't get into the recommendations for what to choose.

For each of the two activations you calculate a separate "style matrix" - so one for your style source image, and the other for your generated image.

To get a style matrix you basically get the correlation (not the mathematically precise term, it's actually the sum of an elementwise multiplication, but "correlation" is supposedly the intuition) between each pairing of channels in the activation volume. So you'd get one number for channels 1 and 2, one for channels 1 and 3, etc. Each time you get the sum of the elementwise multiplication, then, you have a single number which reflects the "correlation" between those channels in the activation volume. You get all such numbers, from all pairings of channels, and put them in a matrix where the x axis is "channel A" and the y axis is "channel B" of each pairing.

So you get one style matrix for your style source image, and one for your generated image. To get your style cost, you just subtract one matrix from the other and get the squared Frobenius norm of that (= you just sum up each of the squared values in the matrix you got from subtracting). This is your style cost.

So obviously, the closer your two style matrices are to each other, the lower your style cost. The style matrix of your style source image won't change since it's just a fixed input (and you aren't training your neural net - the parameters are fixed), so you basically you want the style matrix of your generated image to conform to it. You gradually modify pixel values in the generated image until this occurs.

So the question is, how do you get a generated image that has a similar style matrix? It seems that you want your generated image to activate the same pairings of filters as your style source image. So if the style source image frequently activates filters 2 and 6 at the same time, you want your generated image to do the same thing. The reverse is also true: If the style source image doesn't activate filters 2 and 6 at the same time, your generated image should not do so.

What I don't quite see is: How/why does this quantify style? I guess we could say something like "Artist X tends to use these two motifs in combination, but not these other motifs." Is that why we're doing this? Just for the sake of argument, why not just compare the activations of the two images at that layer? Is it because a straight comparison of activations represents content? Is it simply too simplistic for something like "style"? But suddenly when we're comparing pairs of activations, now it's sufficiently complex to be style?

EDIT: Actually, the "content cost" is in fact generated by subtracting the activations of the generated image (at a particular layer) from the activations of the content image at the same layer (then getting an elementwise sum of squares of the result). So maybe we can say that "content" is defined by activating individual filters, but "style" is defined by activating pairings of filters. I'm not totally sold on my own explanation though.

$\endgroup$
1
$\begingroup$

Just for the sake of argument, why not just compare the activations of the two images at that layer? Is it because a straight comparison of activations represents content? Is it simply too simplistic for something like "style"? But suddenly when we're comparing pairs of activations, now it's sufficiently complex to be style?

Suppose you have a layer with 16 channels of 1000x1000 activations. The "straight comparison" would be norm of differences of 16 million of activations.

The style cost first calculates the correlations in pairs of layers. Each pairwise correlation is over the 1 million activations, and you end up with 16x16 correlation matrix, or 256 elements over which you get the norm of differences.

So, what you're suggesting is to lump all channels into one big pile, and just compare the ultimate activations of the layer ignoring the channel assignment of neurons. In contrast, the style transfer recognizes the channels are higher level abstractions of features themselves, then you see how the channels are correlated with each other. You seem to be focusing on the fact that correlations are calculated pair-wise, but that's not essential. Just like in a correlation matrix in statistics each cell is a pair-wise correlation, here you have pair-wise channel correlation. However, in both cases it is the matrix in its entirety that represents the correlation structure of your data.

My analogy is with correlation analysis in pedometers. Suppose you have several sensors in the pedometer such as gyro, inertial etc. You get a sample of 1 minute length at 100Hz sampling rate. Let's say we have 10 sensors in this pedometer. This means that we get 10 time series of 6,000 observations of each sensor. We could obtain the 10x10 correlation matrix of these sensors. We could obtain another sample of the same length, and get another correlation matrix. It's possible that thee correlation matrices would reflect the mode of movement of the person wearing the pedometer, such as walking, sitting and running. We could also, identify different 10x10 matrices that correspond to different modes.

So, if we applied what you suggested instead we'd be comparing a big pile of 60,000 observations. How likely these to be informative of the mode of movement? Unlikely. You can keep walking, and each 1 minute sample would be very different from another.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.