I'm trying to figure out the matrix multiplications for the implementation of a single hidden layer neural net for MNIST digit recognition in Python.

Like the following:

x1                            z1
x2                            z2
 1             h3


I'm using a hidden layer of size 200.

The number of features for the digits is 784.

The number of classes is 10.

Each label is transformed to a vector of length 10 which has a single 1 in the position of the true class and 0 elsewhere.

Between the input and the hidden layer, I'm going to use a 200 by 785 matrix V.

Matrix V: the i, j - entry represents the weight connecting the jth unit in the input layer to the ith unit in the hidden layer. The ith row of V represents the ensemble of weights feeding into the ith hidden unit.

Between the hidden the the output layer, I'm going to apply a matrix W, which is 10 by 201.

Matrix W: the i, j - entry represents the weight connecting the jth unit in the hidden layer to the ith unit in the output layer. The ith row of W is the emsemble of weights feeding into the ith output unit.

So I start with the input matrix, which is n by 784. Can someone explain what to do? What do I need to multiply it by, and then what/how do I multiply the result by? I'm not sure how exactly to multiply these matrices.

(Let's just call the activation functions f().)

I'm a bit confused by the dimensions of the matrices and not sure when /where/ how exactly to use V and W.

  • $\begingroup$ Are you asking about fitting the model, or about how predictions are made once the weights have been estimated? $\endgroup$ – shadowtalker Apr 24 '16 at 10:32
  • $\begingroup$ I'm asking about fitting the model. @ssdecontrol $\endgroup$ – ajfbiw.s Apr 24 '16 at 18:47

I assume you are just asking how to perform a feedforward pass.

Let's say your input matrix is X [n by 784]

  1. add a column of ones on the left of X to make it [n by 785] for the biases
  2. Hidden weighted input are Z = X*V' [n by 200]
  3. Apply non-linearity to all elements of H to get hidden activations A = f(Z)
  4. Get output as f(A*W') [n by 10] where this last f is probably a softmax output layer computation to get posterior probabilites for every class
  • $\begingroup$ So your V' and W' are V transpose and W transpose, right? Yes I was indeed just asking about the forward pass. $\endgroup$ – ajfbiw.s Apr 24 '16 at 17:29
  • $\begingroup$ Yes exactly, they are the transpose of V and W $\endgroup$ – Steve3nto Apr 25 '16 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.