# intuition behind style matrix in neural style transfer

I guess this is kind of an offbeat question. This is a situation where I understand the "how" and not the "why". Since this may not be well known, I'll provide an explanation first.

The question is concerning the calculation of the style cost when doing neural style transfer between two images (to combine one image which provides content with another that provides style, like making a photo you snapped of your dog look like a Picasso painting). It's a machine learning technique and pretty neat.

To generate this image, you need a content cost (how similar your resulting image is to your "content" image, in this case the picture of your dog) and a style cost (how similar your resulting image is to the Picasso painting). I won't describe the content cost here, only the style cost. Your overall cost is just the sum of the two, though.

To get the style cost, you are supposed to run both images through a pretrained neural network and look at the activations of a particular layer. You can choose the layer you want, I won't get into the recommendations for what to choose.

For each of the two activations you calculate a separate "style matrix" - so one for your style source image, and the other for your generated image.

To get a style matrix you basically get the correlation (not the mathematically precise term, it's actually the sum of an elementwise multiplication, but "correlation" is supposedly the intuition) between each pairing of channels in the activation volume. So you'd get one number for channels 1 and 2, one for channels 1 and 3, etc. Each time you get the sum of the elementwise multiplication, then, you have a single number which reflects the "correlation" between those channels in the activation volume. You get all such numbers, from all pairings of channels, and put them in a matrix where the x axis is "channel A" and the y axis is "channel B" of each pairing.

So you get one style matrix for your style source image, and one for your generated image. To get your style cost, you just subtract one matrix from the other and get the squared Frobenius norm of that (= you just sum up each of the squared values in the matrix you got from subtracting). This is your style cost.

So obviously, the closer your two style matrices are to each other, the lower your style cost. The style matrix of your style source image won't change since it's just a fixed input (and you aren't training your neural net - the parameters are fixed), so you basically you want the style matrix of your generated image to conform to it. You gradually modify pixel values in the generated image until this occurs.

So the question is, how do you get a generated image that has a similar style matrix? It seems that you want your generated image to activate the same pairings of filters as your style source image. So if the style source image frequently activates filters 2 and 6 at the same time, you want your generated image to do the same thing. The reverse is also true: If the style source image doesn't activate filters 2 and 6 at the same time, your generated image should not do so.

What I don't quite see is: How/why does this quantify style? I guess we could say something like "Artist X tends to use these two motifs in combination, but not these other motifs." Is that why we're doing this? Just for the sake of argument, why not just compare the activations of the two images at that layer? Is it because a straight comparison of activations represents content? Is it simply too simplistic for something like "style"? But suddenly when we're comparing pairs of activations, now it's sufficiently complex to be style?

EDIT: Actually, the "content cost" is in fact generated by subtracting the activations of the generated image (at a particular layer) from the activations of the content image at the same layer (then getting an elementwise sum of squares of the result). So maybe we can say that "content" is defined by activating individual filters, but "style" is defined by activating pairings of filters. I'm not totally sold on my own explanation though.