7
$\begingroup$

I'm using two 3-layer feedforward multi-layer perceptrons (MLPs). With the same input data (14 input neurons), I do one classification (true/false), and one regression (if true, "how much")¹. Until now, I've lazily used Matlabs patternnet and fitnet, respectively. Lazily, because I haven't taken the time to really understand what's going on — and I should. Moreover, I need to make the transition to an OSS library (probably FANN), that will likely require more manual setup than the Matlab NN Toolbox. Therefore, I'm trying to understand more precisely what's going on.

The networks created by patternnet and fitnet are nearly identical: 14 inputs neurons, 11 hidden neurons, 1 target neuron (2 for the fitnet, but only 1 piece of informatio). But, they're not completely identical. The differences by default are:

Should those differences be?

What kind of backpropagation functions are optimal for classification, and what kind for regression, and why?

What kind of transfer functions are optimal for classification, and what kind for regression, and why?


¹The classification is for "cloudy" or "cloud-free" (2 complementary targets), the regression is for quantifying "how much cloud" (1 target).

$\endgroup$

migrated from scicomp.stackexchange.com Dec 10 '13 at 14:02

This question came from our site for scientists using computers to solve scientific problems.

2
$\begingroup$

The key difference is in the training criterion. A least squares training criterion is often used for regression as this gives (penalised) maximum likelihood estimation of the model parameters assuming Gaussian noise corrupting the response (target) variable. For classification problems it is common to use a cross-entropy training criterion, to give maximum likelihood estimation assuming a Bernoilli or multinomial loss. Either way, the model outputs can be interpreted as estimate of the probability of class membership, but it is common to use logistic or softmax activation functions in the output layer so the outputs are constrained to lie between 0 and 1 and to sum to 1. If you use the tanh function, you can just remap these onto probabilities by adding one and dividing by two (but it is otherwise the same). tanh is a good choice for hidden layer activation functions.

The difference between scaled conjugate gradients and Levenberg-Marquardt are likely to be fairly minor in terms of generalisation performance.

I would strongly recommend the NETLAB toolbox for MATLAB over MATLABs own neural network toolbox. It is probably a good idea to investigate Bayesian regularisation to avoid over-fitting (Chris Bishop's book is well worth reading and most of it is covered in the NETLAB toolbox).

$\endgroup$
  • $\begingroup$ Interesting — why do you strongly recommend NETLAB over MATLABs own NN toolbox? I've been quite satisfied with the latter, but I'm aiming to move away to become free of commercial licenses. Netlab is clearly still Matlab, so solves only half the problem as far as independence is concerned; I'd preferably use something I can use with Python. $\endgroup$ – gerrit Dec 10 '13 at 17:09
  • $\begingroup$ I suspect NETLAB also works with octave, which solves the commercial license issue. I prefer NETLAB as it has the basic tools for classification and regression set up quite straightforwardly and encourages the use of regularised networks, which is very important in avoiding over-fitting. It also complements Chris Bishop's book very well, and that is the book I'd recommend to anybody who wanted to used neural networks in anger. Gaussian processes are a more modern option though, the GPML toolbox and book are also highly recommended (gaussianprocess.org/gpml). $\endgroup$ – Dikran Marsupial Dec 10 '13 at 18:21
  • $\begingroup$ Ok, I see. It always worked quite well for me, I haven't had over-fitting problems with early stopping. I see the point with Octave but if I anyway need to make a transition, I'd rather move to Python straight away (I applaud Octave, and I wish I could run my Matlab code in it, which relies heavily on post-R2008a classes; but I don't love Matlab as a language enough to choose a FOSS Matlab clone over Octave. Anyway, this discussion is getting off-topic). $\endgroup$ – gerrit Dec 10 '13 at 20:39
  • $\begingroup$ been meaning to learn python myself for a fair while, it is used quite a lot in the machine learning community, so I expect there are some good libraries available. $\endgroup$ – Dikran Marsupial Dec 11 '13 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.