258 views

### Is backpropagation required for optimizing MLPs? [duplicate]

In a neural network (MLP), we can define the error function during training the difference between the target output and the output for the current input. Now we use backpropagation like so: ...
285 views

### Can you solve non-linear problems with a neural network without using backpropagation? [duplicate]

E.g. is it possible to solve the XOR problem without backpropagation. If so, what would a solution look like?
20k views

### Understanding “almost all local minimum have very similar function value to the global optimum”

In a recent blog post by Rong Ge, it was said that: It is believed that for many problems including learning deep nets, almost all local minimum have very similar function value to the global ...
37k views

### Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
3k views

### Why are optimization algorithms defined in terms of other optimization problems?

I am doing some research on optimization techniques for machine learning, but I am surprised to find large numbers of optimization algorithms are defined in terms of other optimization problems. I ...
4k views

### In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
2k views

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
2k views

### Why Expectation Maximization is important for mixture models?

There are many literature emphasize Expectation Maximization method on mixture models (Mixture of Gaussian, Hidden Markov Model, etc.). Why EM is important? EM is just a way to do optimization and is ...
3k views

### Are there algorithms and tools that can optimize black box functions with black box constraints?

Suppose that we have an objective function $f$ which have as parameters $x_1, x_2$. $f$ is an objective function to be maximized for a given problem.Lets say: $$f(x_1,x_2)=x_1+x_2+E(x_1,x_2)$$ ...
331 views

### Building a model to help me determine parameters of a physical water filter?

I am looking to identify the optimal parameters for a sand water filter (a ratio of coarse sand to fine sand) which has the fastest flow rate with the minimum cloudiness in the effluent water. ...
1k views

### Optimizing a “black box” function: Linear Regression or Bayesian Optimization… what's the difference?

Goal: I have a function $f(x,y)=z$ (two variables for illustration only) which I know almost nothing about--it has a compact domain which I can determine, it is non-negative, and bounded above. My ...
2k views

### What concepts in optimization do I need for machine learning?

I am a Math/CS dual major. As part of my math major, I have the option of taking optimization and mathematical programming classes. I am also interested in machine learning. I know that a lot of ...
4k views

### Using randomized search algorithms to find weights for neural network?

I am currently taking a class in machine learning. I had mentioned to a coworker that we were learning about randomized optimization, specifically randomized hill climbing (RHC). He said that it was ...