I am teaching myself about NNs for a summer research project by following an MLP tutorial which classifies the MNIST handwriting database.

I want to change the MLP from classification to regression to understand more about the structure of the network.


The original training data was a $(28,28)$ image array vs. 10 labels

Assume it is now 28 timeseries of length 28 from $t=-27$ to $t=0$ that are assumed to contribute to the value of a timeseries at $t=1$ (any better examples would be welcome!)

Current MLP Structure

Currently the structure of my MLP is as follows:

  1. Input Layer $28^2$ = 728
  2. Hidden Layer = 500
  3. Output Layer = 10
  4. Logistic Regression Layer = (softmax then argmax)
  5. Classification = one of 10 digits


I would like try to implement two difference approaches:

  1. Output a PDF of possible real values
  2. Output an exact real value estimate

From a high level, what should I change in my structure to alter my classifier MLP to a regression MLP?



This paper seems to suggest the following structure for learning time series i.e. a real value regression problem:

  1. Input Layer $28^2$ = 728
  2. Hidden Layer = 500
  3. $b_{pj}$ as a bias vector(?)
  4. Linear transformation (should this just be $y=Wx+B$ ?)
  5. Output Layer = 1 real value

enter image description here

  • $\begingroup$ Use pybrain: github.com/pybrain/pybrain If you really want regression, your output layer should have one output, and should has softmax function. $\endgroup$
    – 404pio
    Commented Apr 30, 2015 at 6:59
  • 1
    $\begingroup$ Thanks for the comment, pybrain seems a bit high level... I want to properly understand what is going on in the networks and be able to build them myself and optimise them using the theano library for GPU acceleration $\endgroup$ Commented Apr 30, 2015 at 20:00

1 Answer 1


One simple example, predict house prices at some location. There will be some factors associated with house prices; number of rooms, house size, age etc. These will be your features. Number of inputs to NN is equal to number of features (784 in MNIST). In the simplest case, you can turn classification into regression just by removing activation function at the output layer (make it linear) and changing cost function to quatratic cost function. Use one output unit because the network predict only single output (price of a house). One hidden layer for starting will be proper. Number of hidden units depends on number of features and complexity of problem. As you backpropagate error and update weights, the network will converge to optimum values. I suggest you to complete Andrew Ng's course in Coursera. It covers various tecniques.

  • $\begingroup$ Thanks for the answer, I think I will use MSE for the cost function and rescale the input and output nodes to fit my data. $\endgroup$ Commented May 4, 2015 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.