Guidelines to improve a convolutional neural network? I am trying to use a convolutional neural network (implemented with keras) to solve a modified version of the MNIST classification problem (I am trying the background variations as described here). I started from this example and played around a bit with the parameters to get better accuracies, but I seem to get stuck at about 90% accuracy on my validation set. 
I've read papers who manage to get near-human accuracy on those datasets, but I seem not to be able to improve my network to get over 95% (something I would expect to be possible). Because I have only been guessing the parameters for the network thus far and I don't seem to find anything online, I was wondering whether there are any guidelines to find a good architecture and good parameters for convolutional neural networks.
Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?
 A: 
Would anybody be aware of how to tackle the problem of finding good parameters (including architecture) for a CNN apart from trying?

No. Hence, they are optimized by 'graduate student descent' :)
A standard-ish architecture for mnist is lenet-5, and closely related variants, eg Karpathy's convnetjs implementation, which uses the following layers:
type:'input', out_sx:24, out_sy:24, out_depth:1
type:'conv', sx:5, filters:8, stride:1, pad:2, activation:'relu'
type:'pool', sx:2, stride:2
type:'conv', sx:5, filters:16, stride:1, pad:2, activation:'relu'
type:'pool', sx:3, stride:3
type:'softmax', num_classes:10

(and I think it augments the data by cutting random 24x24 patches out of the original 28x28 images, which is a key part of obtaining higher accuracies on this (tiny) dataset).
There are meta-learning techniques, which is an open research area. For example "Neural architecture search with reinforcement learning", by Barret Zoph and Quoc Le, 2016, uses reinforcement learning to try different architectures, find out what works well. It does this in an automated way, without human intervention. Of course this needs a ton of GPU power...
A: You might decorrelate your data first using PCA, and then clamp the objects to your input nodes (i.e., input the PCs from PCA into the CNN).  Did you select any features, or use everything?  (don't know if the features were pre-selected and users are expected to use everything?).  
A: I would suggest you try another data set. MNIST data has been "over tuned" on test data set!!
You can try to run a test for "human accuracy" on this data, and you may not get over 95% accuracy. BTW, I tried, there are many digits are not quite recognizable. Here is an example, it can be 3 or 5.

In sum, today's NN tools are really good, that only needs few changes you may get very good results especially on classic data set. My suggestions to you would not be push the limit of the model on MINIST data but try more other data sets.
