1
$\begingroup$

I am new to theano and learning.

Am I doing regularisation correctly? Is this term the correct L2 norm?

0.01 * (wh ** 2).sum() + 0.01 * (wy ** 2).sum() 

Any comment and correction to that code also very welcome, thank you all!

import theano
import theano.tensor as T
import numpy as np
rng = np.random

ninputs = 100
nfeatures = 784
noutputs = 10
nhiddens = 100

#training rate
tr_rate = 0.1 
# number of training steps
training_steps = 100

D = (rng.randn(ninputs, nfeatures), rng.randint(size=ninputs, low=0, high=noutputs))

# Feed-forward
rng = np.random.RandomState(0)
x = T.dmatrix('x')
t = T.ivector('t')

wh = theano.shared(rng.normal(0, 1, (nfeatures, nhiddens)), borrow=True)
bh = theano.shared(np.zeros(nhiddens), borrow=True)
h =  T.nnet.sigmoid(T.dot(x, wh) + bh)

wy = theano.shared(rng.normal(0, 1, (nhiddens, noutputs)), borrow = True)
by = theano.shared(np.zeros(noutputs), borrow=True)
y =  T.nnet.softmax(T.dot(h, wy) + by)

prediction = T.argmax(y, axis=1)

parameters = [wh, bh, wy, by]

# Backprop 
# xent = -t * T.log(y) - (1-t) * T.log(1-y)         # Cross-entropy loss function
xent = -T.mean(T.log(y)[T.arange(t.shape[0]), t])
cost = xent.mean() + 0.01 * (wh ** 2).sum() + 0.01 * (wy ** 2).sum()         # Regularisation

gparameters = [T.grad(cost, param) for param in parameters] # Compute the gradient of the cost    

# Compile
train = theano.function(
          inputs=[x,t],
          outputs=[prediction, xent],
          updates=[(param, param - tr_rate * gparam) for param, gparam in zip(parameters, gparameters)],
          allow_input_downcast=True)

predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    pred, err = train(D[0], D[1])

result = predict(D[0]) - D[1]

N = ninputs

error = 0
for index in range(N):
    if result[index] != 0:
        error += 1

correct_guesses = N - error
accuracy = (N - error)*100/N

print
print "correct predictions = %f over %i examples" % (correct_guesses, N)
print "accuracy = %i%%" % accuracy
$\endgroup$
  • $\begingroup$ This probably belongs on Code Review, not here. If you want to ask about NN's outside of any particular implementation, please clarify your question. $\endgroup$ – gung - Reinstate Monica Dec 18 '15 at 18:30
  • $\begingroup$ I edited my question to fit the rules better. I am mainly concerned about regularisation, and I want to know that line is correct. Thank you also for pointing me to the Code Review section. $\endgroup$ – user Dec 18 '15 at 21:22
  • $\begingroup$ Asking if a particular formula for the L2 norm is correct would be on topic here. Unfortunately, I can't read theano. Can you translate that into mathematical notation? $\endgroup$ – gung - Reinstate Monica Dec 18 '15 at 21:28
1
$\begingroup$

Try to look at this working example http://deeplearning.net/tutorial/mlp.html and this question Is Theano using back-propagation?

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I read the first example and wanted to create a simpler one in a few lines to make it even simpler to follow. I also read the second link and wanted to make sure I had got it right. $\endgroup$ – user Dec 17 '15 at 14:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.