How to deal with changing video frame sizes in a CNN? How to deal with videos where the frame sizes are not the same frame to frame?
For example this video moves up and down and when it does, the video part of the screen has a different amount of pixels vertically.  
How to deal with different frame sizes in a CNN?
 A: I think a better question would be: SHOULD you deal with different video frame sizes. First of all, the video linked has the same frame size throughout the video. It has just been padded with a black border. This shouldn't be a problem for your CNN. 
If you have trained a CNN well enough to detect faces for example, it doesn't matter that there is a black padded border around the 'active' part of video. It won't change the ability of face recognition.
A: The amount of pixels do not change on the video, however, the section that you are interested on at each frame does.
One way you could deal with this is by using image segmentation techniques, this is, at each time-step you can extract the image from the background and feed it to the CNN. If the number of pixels are not exactly the same at each timestamp after segmenting, you can simply re-scale the images to a fixed shape.
In any case, a CNN could easily deal with these type of raw images as they are, due to the translation-invariance property given by the convolution + pooling steps. Think about it as the same way in which a CNN can recognize a cat independently on the position at which it is located on an image.
