How to use convolutional neural nets on 'cylindrical images' Suppose I have a data-set of images that 'live on a cylinder' (for example, if my images were taken by panorama shots that wrap around onto themselves). How would I modify a convolutional neural network to take this into account? A few simple 'hacks' I can imagine performing (and why I consider them not as full solutions):


*

*Currently, I am representing the cylindrical image as a rectangular image (with a cut along the cylinder), and then 'cylindrically padding' by taking the leftmost few columns and adding them to the right:

the idea being that now the network has access to the information that the star is near the 1, but this doesn't 'actually work' in my mind, like it might work for the first convolutional layer, but if I have multiple layers it seems that I actually want my network architecture to reflect the cylindrical symmetry

*Do the same rectangular representation as above, but augment my data-set by taking each image and rotating it about the cylindrical axis pixel by pixel. Again, this feels like a hack and not a real solution


I'm using Keras, and am mostly interested in 'practical solutions', I would have thought Keras would have an option to deal with this situation (maybe like cylindrical padding or something) but I can't find any. The next idea I had in my mind was to write my own Keras layers to deal with this problem (modifying the existing convolutional/pooling layers) but I'm not sure how to start
 A: I'm not sure that there is an exact answer to this question.  That said I find it interesting so here goes:
Having messed around with CNNs myself I think that I would experiment with two main strategies.


*

*Revolved: (I think that this might be your second idea, but I'm not positive from your description) As a form of augmentation take a 360 degree image but randomly shift the vertical divider to somewhere else in the image. For example, if by default the vertical edge is at 0 degrees then randomly select a new degree (0 - 359, I guess) that is the "divider".  Crop all data from the left of the divider and tack it onto the right side. With this done randomly the CNN has a chance to potentially see 360 degrees "smoothly" with your training data but once it's trying to predict it's going to have to survive a divided image. (unless there's a weird mobius like image solution)

*Skidmark: Duplicate the image as some kind of multiplier of the starting width. With some experimentation make the image wider than it was before so the CNN can train on the 'split' area (ie 0 degrees in the previous example) of the image without being interrupted. Depending on how deep your CNN is the features can actually get pretty big so the expansion ratio might need to be relatively high, like 1.3 - 1.6x of initial width. I think that you could estimate the required extra buffer size by taking the largest filter in the first convolutional layer and multiplying by whatever pooling until you're at the last pool.

*Finally pie in the sky but maybe ultimately perfect since there would be less distortion would be potentially some kind of 3d CNN where the volume contains the cylinder. Then using 3d filters train the CNN to predict whatever it is that you're after.
It depends on what you are trying to predict with your model and what kinds of latent features might be in the data that your model is training on and attempting to predict.
Generally I think that I'd try 1 because I think that if anything the moving split would simply help regularize your model. Also it would be computationally cheaper than the other options. Finally it also has the benefit of being able to use fairly typical CNN's as opposed to having to develop your own.
Good luck!
