Hot answers tagged neural-networks
27
I realize this question has been answered, but i don't think the extant answer really engages the question beyond pointing to a link generally related to the question's subject matter. In particular, the link describes one technique for programmatic network configuration, but that is not a "[a] standard and accepted method" for network configuration.
By ...
17
The advice I would give is as follows:
(i) Exhaust the possibilities of linear models (e.g. logistic regression) before going on to neural nets, especially if you have many features and not too many observations. For many problems a Neural Net does not out-perform simple linear classifiers, and the only way to find out if your problem is in this category ...
15
Feed-forward ANNs allow signals to travel one way only: from input to output. There are no feedback (loops); i.e., the output of any layer does not affect that same layer. Feed-forward ANNs tend to be straightforward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organisation is also referred to as ...
14
Neural networks have been around for a while, and they've changed dramatically over the years. If you only poke around on the web, you might end up with the impression that "neural network" means multi-layer feedforward network trained with back-propagation. Or, you might come across any of the dozens of rarely used, bizarrely named models and conclude that ...
13
A Bayesian network is a type of graphical model. The other "big" type of graphical model is a Markov Random Field (MRF). Graphical models are used for inference, estimation and in general, to model the world.
The term hierarchical model is used to mean many things in different areas.
While neural networks come with "graphs" they generally don't encode ...
12
The back propagation algorithm is a gradient descent algorithm for fitting a neural network model. (as mentionned by @Dikran) Let me explain how.
Formally: Using the calculation of the gradient at the end of this post within equation [1] below (that is a definition of the gradient descent) gives the back propagation algorithm as a particular case of the ...
12
First of all RBM's are certainly different from normal Neural Nets, and when used properly they achieve much better performance. Also, training a few layers of a RBM, and then using the found weights as a starting point for a Mulitlayer NN often yields better results than simply using a Multilayer NN.
The best pointer I can think of is this course on ...
11
What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this approach: events in the inputs can only be correlatd with other inputs/outputs which lie at most t timesteps apart, where t is the size of the window.
E.g. you can ...
11
The answer to your question is in my experience "no", SVMs are not definitely superior, and which works best depends on the nature of the dataset at hand and on the relative skill of the operator with each set of tools. In general SVMs are good because the training algorithm is efficient, and it has a regularisation parameter, which forces you to think ...
11
You are right about unlabeled data. RBMs are generative models and most commonly used as unsupervised learners.
When used for constructing a Deep Belief Network the most typical procedure is to simply train each each new RBM one at a time as they are stacked on top of each other. So contrastive divergence isn't going up and down in the sense that I think ...
10
Here is a simple recipe that may help you get started writing code and testing ideas...
Let's assume you have monthly data recorded over several years, so you have 36 values. Let's also assume that you only care about predicting one month (value) in advance.
Exploratory data analysis: Apply some of the traditional time series analysis methods to estimate ...
9
I will just try to explain my opinion that appeared to be shared by most of my friends. I have the following concerns about NN that are not about SVM at all:
In a classic NN, the amount of parameters is enormously high. Let's say you have the vectors of the length 100 you want to classify into two classes. One hidden layer of the same size as an input ...
8
Estimate feature importance by randomly bumping every value of a single feature, and recording how your overall fitness function degrades.
So if your first feature $x_{1,i}$ is continuously-valued and scaled to $[0,1]$, then you might add $rand(0,1)-0.5$ to each training example's value for the first feature. Then look for how much your $R^2$ decreases. ...
8
How about anomaly detection algorithms? As you mention Andrew Ng's class you'd probably seen the "XV. ANOMALY DETECTION" section on ml-class.org, but anyway.
Anomaly detection will be superior to a supervised classification in scenarios similar to yours because:
normally you have very few anomalies (ie., too little "positive" examples)
normally you have ...
8
In R you can use neuralnet and RSNNS (which provides an interface to the Stuttgart Neural Network Simulator) to fit standard multilayer neural networks, but there are differences to RBM.
Regarding implementing deep neural nets in R, I think the only worthwhile strategy would be to interface existing FOSS implementations, which is usually a much better ...
8
Symmetry breaking.
If all weights start with equal values and if the solution requires that unequal weights be developed, the system can never learn.
This is because error is propagated back through the weights in proportion to the values of the weights. This means that all hidden units connected directly to the output units will get identical error ...
7
Back-propogation is a way of working out the derivative of the error function with respect to the weights, so that the model can be trained by gradient descent optimisation methods - it is basically just the application of the "chain rule". There isn't really much more to it than that, so if you are comfortable with calculus that is basically the best way ...
7
The no-free lunch theorems suggest there is no classifier that is a-priori superior to any other, and the choice of classifier depends on the nature of the particular dataset. I wouldn't comit myself to a choice of classifier and would instead evaluate several methods.
The classes are only mildly imbalanced, so I suspect that shouldn't be a key factor in ...
7
The streaming setting in machine learning is called "online learning". There is no exact support vector machine in the online setting (since the definition of the objective function is inherently for the batch setting). Probably the most straightforward generalization of the SVM to the online setting are passive-aggressive algorithms. Code is here ...
7
It is a matter of trade-offs. SVMs are in right now, NNs used to be in. You'll find a rising number of papers that claim Random Forests, Probabilistic Graphic Models or Nonparametric Bayesian methods are in. Someone should publish a forecasting model in the Annals of Improbable Research on what models will be considered hip.
Having said that for many ...
7
There are severe flaws with this approach.
First, there are many gambles which usually win, but which are bad gambles. Suppose you have the chance to win \$1 $90\%$ of the time and lose \$100 $10\%$ of the time. This has a negative expected value, but the way you are training the neural network would teach it to recommend such reverse lottery tickets.
...
6
Here's a good quick introduction:
intro to neural networks.
Note that R has neural-network functionality, so no need to spend any time implementing NN yourself until you've given it a spin and decided it looks promising for your application.
Neural networks are not obsolete, but they have gone through a couple of hype cycles, and then after realizing they ...
6
Andrew Ng's cs294a course at Stanford Deep Learning and Unsupervised Feature Learning has some nice freebies on sparse autoencoders:
http://www.stanford.edu/class/cs294a/video1.html
http://www.stanford.edu/class/cs294a/video2.html
http://www.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf
See at the bottom of: ...
6
In most cases if you remove unnecesary connections you'll get better network. It is easy to overtrain (overfit) the network --- in which case it will perform poorly on validation dataset.
Pruning unnecesary connections will most probably reduce o overtraining probability. Please see: http://en.wikipedia.org/wiki/Overfitting .
6
As a rule of thumb, small and/or sparse networks generalise better. You can let your training algorithm weed out unecessary connections within a fixed-size network by applying some form of weight decay, or you can apply an algorithm that aims to optimise network architecture/topology itself through removing unecessary inputs, hidden nodes or connections.
...
6
This has been researched for 20 years or so, and there are many papers claiming to outperform backpropagation. Xin Yao did a lot of work on this in the 1990s, and Kenneth Stanley created one of the currently most active frameworks, NEAT (NeuroEvolution of Augmenting Topologies (see http://www.cs.ucf.edu/~kstanley/neat.html and ...
6
One standard approach is to use a restricted Boltzmann machine to do the feature extraction, and then reconsider the RBM as a neural network and finish the training using back-propagation. See, for example,
G. E. Hinton, "To Recognize Shapes, First Learn to Generate images," Progress in brain research, vol. 165, pp. 535-547, 2007.
This is an example of ...
6
No, there is no need to worry about this, because the non-linear transformation means that the feature space generated by the hidden layer neurons can be of higher dimension than the input space without being linearly dependent.
Consider Ripley's synthetic benchmark dataset, which consists of two classes, each of which is represented by two Gaussian ...
6
I highly recommend watching these lectures and use this as reading material. These lectures are on machine learning in general by Andrew Ng talks in length about neural networks and does try hard to make it accessible for beginners.
Only top voted, non community-wiki answers of a minimum length are eligible
