Linked Questions

0
votes
1answer
258 views

Is backpropagation required for optimizing MLPs? [duplicate]

In a neural network (MLP), we can define the error function during training the difference between the target output and the output for the current input. Now we use backpropagation like so: ...
0
votes
1answer
285 views

Can you solve non-linear problems with a neural network without using backpropagation? [duplicate]

E.g. is it possible to solve the XOR problem without backpropagation. If so, what would a solution look like?
49
votes
1answer
20k views

Understanding “almost all local minimum have very similar function value to the global optimum”

In a recent blog post by Rong Ge, it was said that: It is believed that for many problems including learning deep nets, almost all local minimum have very similar function value to the global ...
36
votes
5answers
37k views

Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
23
votes
4answers
3k views

Why are optimization algorithms defined in terms of other optimization problems?

I am doing some research on optimization techniques for machine learning, but I am surprised to find large numbers of optimization algorithms are defined in terms of other optimization problems. I ...
21
votes
2answers
4k views

In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
9
votes
4answers
2k views

Gradient descent optimization

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
15
votes
2answers
2k views

Why Expectation Maximization is important for mixture models?

There are many literature emphasize Expectation Maximization method on mixture models (Mixture of Gaussian, Hidden Markov Model, etc.). Why EM is important? EM is just a way to do optimization and is ...
5
votes
3answers
3k views

Are there algorithms and tools that can optimize black box functions with black box constraints?

Suppose that we have an objective function $f$ which have as parameters $x_1, x_2$. $f$ is an objective function to be maximized for a given problem.Lets say: $$f(x_1,x_2)=x_1+x_2+E(x_1,x_2)$$ ...
3
votes
3answers
331 views

Building a model to help me determine parameters of a physical water filter?

I am looking to identify the optimal parameters for a sand water filter (a ratio of coarse sand to fine sand) which has the fastest flow rate with the minimum cloudiness in the effluent water. ...
5
votes
1answer
1k views

Optimizing a “black box” function: Linear Regression or Bayesian Optimization… what's the difference?

Goal: I have a function $f(x,y)=z$ (two variables for illustration only) which I know almost nothing about--it has a compact domain which I can determine, it is non-negative, and bounded above. My ...
2
votes
1answer
2k views

What concepts in optimization do I need for machine learning?

I am a Math/CS dual major. As part of my math major, I have the option of taking optimization and mathematical programming classes. I am also interested in machine learning. I know that a lot of ...
1
vote
2answers
4k views

Using randomized search algorithms to find weights for neural network?

I am currently taking a class in machine learning. I had mentioned to a coworker that we were learning about randomized optimization, specifically randomized hill climbing (RHC). He said that it was ...
0
votes
0answers
990 views

Using particle swarm optimization (PSO) h in neural network training?

Does using PSO have advantages/disadvantages over back-propagation when training neural networks? Please give your opinion if you have used PSO or other heuristic methods.
1
vote
1answer
570 views

How were neural network trained before backpropagation was proposed?

When learning neural networks one can often hear that Hinton proposed backpropagation in 1986. After this big leap forward, we could train neural network efficiently. But I have a question: How did ...

15 30 50 per page