2
$\begingroup$

I am interested in knowing about generative methods that generate signals (e.g. images) of varying sizes. But the size generation being sort of "smooth/continuous". So for example, generating images of total size 3000 smoothly up to 196,608 (or even more but every number in between matters to me) with a single model.

The only (what I think is totally ridiculous) idea I had is outputting each signal value (e.g. pixel) via an RNN (since RNNs are size-independent). So that completely solves the issue but my RNN might end up being of length 3000 or 196,608 at times (or hopefully even more)...which seemed ridiculous to me. Is there any real machine learning paper (ideally Deep Learning) that solves such a task?

In fact, are there any image generation tasks with such nature? I am not familiar of any but if anyone knows that would be fantastic! Note, however, that this is not really a transfer learning problem. So have 196,608 version of the output layer seems a bit crazy for me (since I might even want more...). The only made-up task on the spot I can come up with for this is generating all size crops for imagenet data set and then trying to generate any of them. I guess that might be an instance of the task I have in mind. Is there a real/serious task like this that the Machine Learning community is working on?


Cross posted:

$\endgroup$
  • $\begingroup$ How about audio/text generation? For instance, the recent text generator from Openai can generate text on demand with varying size. In general, I would say autoregressive models are one way to go about this. $\endgroup$ – GrigorisG Jul 28 '19 at 10:33
  • $\begingroup$ @GrigorisG varying text imho doesn't solve the problem I want because its symbolic in nature. So generating 20 words/tokens isn't useful (e.g. seq2seq isn't useful to me, or at least its not what I had in mind...). Outputting each pixel over a varying input size & output size is useful. I've been doing a literature search and image segmentation networks seem promising but need to keep reading... $\endgroup$ – Charlie Parker Jul 28 '19 at 14:52
  • $\begingroup$ Why not just learn a generative model of the highest resolution images you need, and then downsample and crop images randomly according to the distribution of image sizes? $\endgroup$ – shimao Jul 28 '19 at 19:30
  • $\begingroup$ @shimao oh the cropping example I made up was just made up! What I truly care is receiving a signal of some size and then processing it (perhaps according to some other inputs) an output vector of any size. As a start it would be nice to return a signal of the same size (so both input and output varying but are the same). This is why image segmentation looked promising to me. $\endgroup$ – Charlie Parker Jul 29 '19 at 15:49
  • $\begingroup$ What does "smooth" mean? There are many "measures of goodness". Some of them allow stunningly simpler implementations than ridiculously long RNN's. How would you personally tell if it was "smooth enough"? What is the input data? Output form? Architecture both hardware and software? Make a good question, and you can get a better answer for your bounty-points. $\endgroup$ – EngrStudent Aug 5 '19 at 12:16
0
$\begingroup$

The approach: fully-convolutional generative models

You could try using a fully-convolutional generative model such as a Variational Autoencoder, which has been used for many image generation tasks. Variational Autoencoders (VAEs) are made of an encoder network which compresses an image to a lower-dimensional Gaussian representation and a decoder network which reconstructs the original image. If you feed noise into the decoder network directly you can generate images.

An Example

Since a convolutional filter can be applied to an image of any size, fully-convolutional models can take in arbitrary images and will produce images with output sizes which are a constant fraction (or constant multiple) of the input image size. To use an absurdly very simple example, imagine you trained a VAE with an encoder made of one convolutional layer and a decoder made of one transposed convolutional layer (each with stride 2). If you generated noise of size MxN and fed it into the decoder half of your VAE, you would get an output of size 2Mx2N.

Producing even smoother outputs

This method along wouldn't produce quite smooth (for instance, the model described would only produce even-width/even-height images). If you care about having every single possible pixel dimension you could add an extra convolutional layer at the end with stride 1 (stride 1 will keep the image output size about the same as the input) and pad your image appropriately before passing it into the layer so the output has the desired size.

I couldn't find any papers using a model like this for the type of image generation task you described, but fully convolutional models have been used successfully for semantic segmentation of variable-sized images.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.