CNN and kernel sizes: is upsampling useful? I am playing with Deep Recurrent Q-Network in Reinforcement learning. 
The architecture I am currently using is similar to the one presented in "Human-level control through deep reinforcement
learning" (Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al.) Nature volume 518, pages 529–533 (26 February 2015):
The input to the neural network consists of an 84x84x3 image produced by the preprocessing map. 
The first hidden layer convolves 32 filters of 8x8 with stride 4 with the input image and applies a rectifier nonlinearity. 
The second hidden layer convolves 64 filters of 4x4 with stride 2, again followed by a rectifier nonlinearity.
This is followed by a third convolutional layer that convolves 64 filters of 3x3 with stride 1 and then a fourth conv layer with 512 filters of 7x7 with stride 1. 
This is the convolutive part of the network: it works! However I thought that this is largely oversized with respect to my problem, a simple grid game where the state is represented by a RGB 11x11 matrix. In fact, the preprocessing function actually upsample the matrix in order to match the input shape of the model. What's the point in resizing the grid from 11x11 to 84x84 (and thus having to manage a way larger set of weights)?
However any manual attempts to define a simpler architecture that I tried is a failure, there's no learning at all!
For example, I tried the following convolutive module (input shape: 11x11x3):


*

*32 filters, 4x4, stride 2

*64 filters, 2x2, stride 1

*512 filters, 3x3, stride 1


I've read many similar questions, but wasn't able to get a hint about this kind of problem (I'm a beginner, so no hyperparameters experience!). Could you offer any insights?
 A: There are several potential problems I can see. First off, you can't use a 4x4 filter of stride 2 on 11x11 input. You won't be able to capture the entire input space (ie the last row and column) and it might be causing the network to fail without an error. 
The reason why is kind of difficult to explain in words so here is a link giving a basic overview of conv nets. (If you're a beginner it would probably be helpful for you to read the all the CS231 course notes actually) If you scroll down a bit there is a moving diagram that will help you visualize why a stride of 2 won't work. 
Also, after briefly looking through the code, I noticed several places where the input dimensions (84 x 84) are hardcoded. There's a possibility that you've missed adjusting the input dimensions somewhere. 
My advice would be to first scan through all the files and make sure you've adjusted all the inputs to from 84 x 84 to 11 x 11. I would also just use 3x3 filters of stride 1 for all layers. You're input space is so small that you don't really need any downsampling (which is why you generally use larger strides). If it still doesn't work, you're going to need to sit down and fully understand what the code is doing at every step in order to debug it. 
A: I agree with Aleksander that 11x11 input and 4x4 filter won't work together. Also if your padding is VALID rather than SAME the following would happen to your model:


*

*Layer 1: 11x11 input, 4x4 kernel, stride 2 -> Output size is 4x4 

*Layer 2: 4x4 input, 2x2 kernel, stride 1 -> Output is 3x3 Layer 3:

*Layer 3: 3x3 input, 3x3 kernel, stride 1 -> Output is a scalar.


This is, at least, very uncommon. You should consider removing one or two layers and matching input and kernel size.
