Can't get Accuracy above 15% on CIFAR-10 dataset I am using Convolutional Neural Networks to tackle image recognition. I used it for MNIST and got an accuracy of 99% but on trying it with CIFAR-10 dataset, I can't get it above 15%. It doesn't seem to learn at all.
I load data in dict, convert the labels to one-hot, then do the following below:
1.) Create a convolution layer with 3 input channels and 200 output channels, do max-pooling and then local response normalization
2.) Create a second convolution layer with 200 input channels and 500 output channels, do max-pooling and the normalization
3.) I use reLu activation except in the final layer which is sigmoid.
4.) I use softmax cross entropy cost function and Adam Optimizer.
def initializer(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def conv2d(x,W):
    return tf.nn.conv2d(x , W , [1,1,1,1] , padding="SAME")

def max_pool(x):
    return tf.nn.max_pool(x , [1,2,2,1] , [1,2,2,1] , padding="SAME")

W_conv1 = initializer([2 , 2 , 3, 200])
b_conv1 = initializer([200])

W_conv2 = initializer([5 , 5 , 200 , 500])
b_conv2 = initializer([500])

W_fc1 = initializer([8*8*500 , 2048])
b_fc1 = initializer([2048])

W_out = initializer([2048 , 10])
b_out = initializer([10])

x_image = tf.reshape((x/255.0) , [-1,32,32,3])

h1 = tf.nn.relu(conv2d(x_image , W_conv1) + b_conv1)
h1 = max_pool(h1)
h1 = tf.nn.local_response_normalization(h1)

h2 = tf.nn.relu(conv2d(h1 , W_conv2) + b_conv2)
h2 = max_pool(h2)
h2 = tf.nn.local_response_normalization(h2)

h2 = tf.reshape(h2 , [-1,8*8*500])

h3 = tf.nn.relu(tf.matmul(h2 , W_fc1) + b_fc1)

h3 = tf.nn.dropout(h3 , keep_prob)

h_out = tf.nn.sigmoid(tf.matmul(h3 , W_out) + b_out)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=h_out))

train = tf.train.AdamOptimizer().minimize(cost)

correct_pred = tf.equal(tf.argmax(h_out, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

This is the gist of my code.
Apart from that, I loaded the data and converted it into one-hot encoding using the below code:
import numpy as np
import cPickle
def unpickle(file):
    with open(file , 'rb') as fo:
        dict = cPickle.load(fo)
    return dict

def convert_one_hot(data,size):
    data_size = np.shape(data)
    final = np.zeros((data_size[0],size))
    final[np.arange(data_size[0]), data] = 1
    return final

def next_batch(dict,size):
    data_size = np.shape(dict["data"])
    label_size = np.shape(dict["labels"])
    final = (data_size[0] , (data_size[1] + label_size[1]))
    data = np.zeros(final)
    data[:,:data_size[1]] = dict["data"]
    data[:,data_size[1]:] = dict["labels"]
    np.random.shuffle(data)
    return data[:size,:data_size[1]], data[:size,data_size[1]:]

Can anyone help me get some accuracy on the dataset?
CIFAR-10 : https://www.cs.toronto.edu/~kriz/cifar.html
Thank you.
 A: Woaa, hold on a second, I haven't ran the code myself, but already know why its not working: you are just exploding the number of model parameters without thinking what they do. So lets check what you are trying to do:


*

*How many parameters do you have?
Just the first FC layer: 8*8*500*2048 = 65 million parameters.
The entire CIFAR-10 dataset is 153 million pixels. You can essentially store half of your dataset in parameters, this is a very bad ratio. Practically you fit one variable for every input, your model is not sufficiently overdetermined no wonder that you are brutally over-fitting. You either need more training data (augmentation) or drastically cut the number of parameters. Add more convolutional/pooling layers to reduce the image dimensions further, but here comes the next catch: 

*What does this convolutional layer do?
W_conv1 = initializer([2 , 2 , 3, 200])
...
h1 = max_pool(h1)

You expect 200 basic combinations of 3x2x2 = 12 pixels. Realistic structures start around 3 pixels, but even then there are no 200 independent bases for a 3 channel input. Either go 256x5x5 kernels or 64x3x3 (the larger the kernel the more independent bases). The max pooling makes it even worse, essentially the first layer does nothing but resizing by local maximum interpolation with some smoothing. You need some overlap between your kernels, so take at least 3x3 kernels.

*For such a small dataset, I find the number of low-level channels generally too high, the 200 and 500 channels for 3 and 7 pixel receptive field, or a 2048 channel FC layer right after a not-so-downsized convolution is a luxury for this dataset and will lead to over-fitting. 
