0
$\begingroup$

What type of models exist to optimize an output based on certain inputs, if you do not know the mathematical relationship between these, but have access to past input and output data?

For example, let's say I am trying to optimize the purity of a chemical compound in an experiment, and I have done hundreds of experiments in the past, for which I have the output (purity) and inputs (concentration of each component, temperature, pressure, etc...).

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ If your output is continuous, this is called regression, there are hundreds of approaches. Alternatively, if the output is discrete it is called classification, with a comparable abundance of approaches. $\endgroup$
    – Marc Claesen
    Sep 9, 2015 at 4:37
  • $\begingroup$ what, exactly, is the constrained optimization part of the question? $\endgroup$
    – Cliff AB
    Sep 9, 2015 at 5:14
  • $\begingroup$ A regression (it is continuous) will fit a function to set a of inputs and outputs. The 2nd step of this process is the constrained optimization of the function (I want the output to be as large as possible, what inputs should I use?). Constraints, for example, are that the concentration of each component has to be between 0 and 1. To rephrase my questions a little bit: Is the way to go about this actually to fit a function, then to find its global maximum within the constraints by using standard calculus techniques, or are there specific models developed to integrate these 2 steps? $\endgroup$
    – Alexis Eggermont
    Sep 9, 2015 at 5:24
  • $\begingroup$ "Constrained multiple regression" is an existing technique that is fairly well-established; look it up and then refine your question if that is what you want/need. Having said that, if you have absolutely no idea what your function (linear regression assumes some kind of additiivity and linearity) you might want to try an "artificial neural network", aka, a universal function approximator. $\endgroup$
    – usεr11852
    Sep 9, 2015 at 8:37
  • $\begingroup$ I thought constrained regression imposed constraints on the coefficients, whereas it is my input and output variables that need to be constrained (not for the regression, but for the maximization). For example in the regression y = bX + e, constrained regression would impose constraints on b. In reality I am trying to find which X (within constraints) will maximize y. I am not looking for shrinkage methods. My question is really if these 2 steps: 1) fit to a function through regression 2) algebraic maximization of that function with constraints, are the right way to go, or if missed something $\endgroup$
    – Alexis Eggermont
    Sep 9, 2015 at 8:49

1 Answer 1

Reset to default
2
$\begingroup$

If new data is expensive/time-consuming to generate, surrogate-model based optimization may be good approach.

That is:

  1. fit a suitable, data-driven model to your data
  2. optimize the surrogate model to derive a new candidate solution
  3. evaluate candidate solution (that is, measure true purity)
  4. update model with the new data you just obtained
  5. repeat 2.-4. until stopping criterion is reached (budget depleted, target purity attained)

What defines a suitable model depends on your problem. For something non-linear, neural networks (as noted by usεr11852), support vector machines or Gaussian Process Regression (aka Kriging) might work. On the other hand, the computational effort for training a model will be adversely affected if your data-set is rather large.

Kriging provides the advantage of giving an uncertainty estimate of its prediction. This allows to compute the Expected Improvement of a candidate solution. While this does not exactly "integrate these 2 steps" (modeling+optimization) EI provides the means to elegantly balance Exploitation vs. Exploration in a surrogate-model based optimization framework.

I suggest reading Engineering Design via Surrogate modeling, Forrester et al. (2008). Since you already have pre-existing data, you can probably skip Chapter 1 (sampling plans), and focus on Chapter 2 and 3.

Regarding your constraints: If the constraints themselves are inexpensive to calculate on-the-fly, just respect them in the above step 2. (as you already suggested in your comment). If constraints themselves are expensive to evaluate, you may consider to replace them with a model, too. Similar to EI, you can use the uncertainty estimate of a Kriging model to compute the probability of feasibility.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.