Optimizing black box functions without Bayesian Optimization I have a black box system which accepts a data instance $\boldsymbol{x}$ and outputs a corresponding $y$. I don not know the exact mathematical formula of the system. What I want to do is maximizing $y$ by tweaking $\boldsymbol{x}$. Since I don't have the exact formula of the system, I cannot use gradient descent to do this. One idea in my mind is to add random noise to $\boldsymbol{x}$ which results in $\hat{\boldsymbol{x}}$, and then sample corresponding $\hat{y}$ from the system. By doing this I could use Bayesian Optimizaton to find an $\hat{\boldsymbol{x}}$ which maximizes $\hat{y}$. 
My question is: 
Is there any other methods to tackle this problem? Can you point me out any paper/book/tutorial?
Thank you
 A: Derivative-free optimization methods solve these type of problems where you can view your objective function as a black box. Bayesian optimization is one type of derivative-free method. It helps if you know any other structural information about your objective function. E.g., can you assume convexity? Surrogate models can used to estimate approximate gradient, e.g., take a few points, fit a quadratic function, and use the model to do a local search in the direction of its negative gradient. There is a vast amount of literature on derivative-free methods (for a review of algorithms, see http://thales.cheme.cmu.edu/dfo/comparison/dfo.pdf). Another example of a derivative-free method is the Nelder-Mead https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method . 
Of course, because very little information is known about the objective function, it is difficult to solve problems with high input dimension (compared to regular optimization problems) -- you would need to consider the number of function evaluations you can afford and the size of your input variable you're optimizing. 
