9
$\begingroup$

I'm an experimental physicist by training and have used standard statistical methods to analyze data, and the design of experiments (DOE) framework to develop models of systems by varying inputs and measurement outputs.

Recently, I've been looking into the use of machine learning and I'm trying to figure if there's any utility/benefit over DOE.

I'm hoping someone on this forum can either validate the way I'm thinking about supervised machine learning, or point out what I'm missing.

I've basically come to the conclusion that supervised machine learning is a method to compute a transfer function of a system given the training data is a set of data that connects the set of inputs with what the output truth should be.

Notwithstanding the machinery that figures out the transfer function based on the training set, what is the difference between DOE and supervised machine learning in terms of the accuracy or other performance measure of the transfer function?

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ The basic idea behind design of experiments is that, by systematically manipulating the values of several variables of interest while controlling all other variables (by holding them constant or randomizing), our statistical models will give us causal knowledge of the relationship between the manipulated and dependent variables. Machine learning methods are focused purely on making accurate predictions, using whatever data are available and without taking into account how they were produced. $\endgroup$
    – Dan Hicks
    Dec 18, 2017 at 2:04
  • 1
    $\begingroup$ Most machine learning "basis engines", such as neural networks, svm, or CART ensembles, are universal function aproximators. My experience with DOE, even RSM, is that the bases are fairly limited and are often polynomial. As a physics guy, you can dislike that, and I think your "transfer function" is looking toward frequency domain models in that sense. Think multivariate, high-interactions, and perhaps neural model-adaptive control. I dispute the "prediction-only" assertion, but accede prediction-dominant use in the market. $\endgroup$
    – EngrStudent
    Dec 18, 2017 at 2:23
  • $\begingroup$ sorry if I missed it, but what exactly is the main difference that is confusing you? $\endgroup$
    – Charlie Parker
    Dec 18, 2017 at 2:27
  • $\begingroup$ Pinocchio What I'm trying to understand is whether or not there is a difference between DOE and supervised machine learning. From the previous comment it would appear that the only difference is the set of basis functions that the transfer function is decomposed upon. Do you agree? $\endgroup$
    – JQK
    Dec 18, 2017 at 12:38
  • $\begingroup$ @JQK I won't lie to you, but I am not familiar (yet) with DOE. I plan to read it in the next days...but supervised learning can be done with any basis functions. Supervised learning is just a paradigm of ML for learning a target function when samples of x and f(x) are known (i.e. the right labels). $\endgroup$
    – Charlie Parker
    Dec 19, 2017 at 2:18

6 Answers 6

Reset to default
6
+100
$\begingroup$

Your question is difficult to answer because there is no "supervised ML algorithm". There are a large number of different ML algorithms that can be optimized in a supervised fashion, each with their strengths and weaknesses.

On a very abstract level, you can define Machine Learning (ML) as a search through some space $P$ for a parameterization ($\theta$) of a given model $M$ such that $M(x;\theta)$ gives a minimal value (though not always globally) for the cost function ($\mathcal{C}$) and input ($x$). More formally:

$$\arg\min_{\theta\in P} \mathcal{C}(M(\theta))$$

For supervised learning, one form the cost function can take is (given $y$ as ground truth):

$$\mathcal{C}(M(x; \theta)) = ||M(x;\theta) - y||_2$$

Any search through $P$ that minimizes the cost function can fit in this framework, and thus you can claim DOE as a ML algorithm if you want. Specifically, an ML algorithm is defined by: the optimization technique employed, the model used, and your cost function. If you fill those in for DOE, you can start to compare it against other ML algorithms.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My question relates to the output of the ML algorithm and not the method that produced it. This, of course, presumes that any supervised ML algorithm creates the similar functionality. That said, my friend Abe tells me that the output of the ML algo is, in fact, a model of the transfer function that maps x to f(x); however, the basis set can be very complicated and isn't necessarily a linear combination of orthonormal functions. $\endgroup$
    – JQK
    Dec 19, 2017 at 18:28
  • $\begingroup$ That's correct. It's generally considered ML if you optimize your model to some data giving you an M_{opt} (M_{opt}(x) = M(x; \theta), there \theta are the parameters you've optimized over). $\endgroup$
    – Ranon
    Dec 21, 2017 at 16:14
7
$\begingroup$

Substitute "supervised ML" with "regression analysis", and you'll see that your question is difficult to answer. Depending on how narrowly you define regression analysis, DOE could be part of the term.

Suppose, you're planning to use regression analysis to determine efficiency of a certain herbicide in the field. One of the very first step would be to design the experiment. In this case I'd argue that DOE is inseparable from your regression analysis, it's part of it.

In many applications of ML you're given the data, and can't do much about planning the experiments, in fact, there is no experiment. For instance, in Kaggle competitions everything's usually set in advance, the training dataset is pre-defined and given, the test data set will be given at evaluation step, and you can't do anything about these things. That is why DOE is not mentioned al ot in the field.

However, it doesn't have to be this way. Suppose, you're building self-driving cars. Yes, as usual, you can employ all the available datasets to train your ML vision components, all the images with traffic signs and road situations, nicely tagged etc. No DOE involved here, really. Yet, once you're out of the initial phase, and get into the field tests, things change. All the typical concerns that DOE addresses show up, e.g. would you need 10 test cars or 1,000 to get reliable results? Would you need 1,000 miles or 1,000,000 miles on the empty roads before trying the test car on a real street? etc.

ML may not necessarily call DOE what they do to plan the development, but in essence it is DOE. Therefore, the answer to your question lies in how narrowly or broadly you define ML term. Is it just fitting the function to the data using the cost (loss)? Or is it a more general definition of building the reliable machine that replaces humans, which would include more than just fitting/optimization but at least some aspects of DOE.

Improve this answer
$\endgroup$
4
$\begingroup$

I just finished a graduate-level course in experimental design and am starting to learn machine learning... I bet there are people on this website who can answer this better than I can, but hopefully this answer will do.

At their cores, experimental design (ED) and machine learning (ML) have different goals. The primary goal of ED is to assess the influences of treatments and, if applicable, compare the influences of different treatments. The primary goal of ML is to give accurate predictions.

These different cores thus influence how each topic is developed:

  • In ED, emphasis is placed on good design so that the variability (of treatment parameter estimates) is reduced, sometimes with the need to meet budgetary constraints. My ED professor once said something to the tune of "Statisticians are always criticized for demanding a large sample size. If you were a statistician working on missiles, firing a missile is a few million bucks down the drain." Fractional factorial designs, from what I understand, are particularly popular due to budgetary constraints. The primary goal of ED is statistical inference of treatment parameters.
  • In ML, emphasis is placed on using predictive algorithms and the issues behind them (e.g., computational compexity, computer software/hardware issues, etc.).

Given my experience in both, I would not try to compare the two subjects. It is like comparing apples to oranges.

Improve this answer
$\endgroup$
1
$\begingroup$

I think that there is a fundamental problem in the way the question is posed. DoE does not per se model a system, the scope of DoE is to efficiently excite the system to gain maximum amount of information with a limited set of resources (time and experiments).

Machine Learning(ML) supervised or unsupervised on the other and is a modelling technique that aims at finding relations within huge amount of observational data already collected. Much like simple regression is the step after you collected the data.

There might be some similarities between DoE and Re-inforced Learning methods, were the ML itself is able to move paramters in the system in order to find an optimal fit or minimize a predifined objective function. In that context the ML model is performin some kind of DoE to gain more information and improve itself.

Improve this answer
$\endgroup$
1
  • $\begingroup$ While it’s true that DOE excited the system, when it does so you’re creating a data set (output, input) that represents the transfer function of the system that is being excited. Isn’t this measured transfer function similar to supervised learning? $\endgroup$
    – JQK
    Jun 20, 2021 at 11:59
0
$\begingroup$

4 years past. hope I can still contribute to your question. I have actual experience in DOE applied into real world of semiconductor manufacturing, more than 30x conducted DOE RSM. And am also a big data computational enthusiast. My direct answer: yes DOE is a supervised ML. DOE is part of ML. I cant say ML is part of DOE. Because ML is a wide topic and method of application. Probably you already knew that DOE RSM is more informative than the other method of DOE such as Taguchi. so stick with DOE RSM (Response surface methodology). a traditional DOE, but very effective. Method of calculation applied to DOE RSM in fractional & full factorial is Matrix Multiplication.(linear algebra). there are other ways to connect and calculate the missing link/values in order to come out, a model. usually Polynomials in 3D. you can apply other models forcibly, such as exponential, weibul, hyperbol, and many more models. How do you judge if you selected the correct model? no worries, use the regression analysis result such as R Sq and RsqAdj and also use Chi sq test, to see if your model fits into your actual data. if model to actual is too far, use other models. if still too far, then review again. Your last question, What is the difference between DOE and supervised machine learning in terms of the accuracy or other performance measure of the transfer function? in terms of accuracy... well this depends on application. if your experiments behaves with the models identified by dead mathematicians, then basically no difference. as long as the DOE and Machine learning calculation applied the same model. otherwise, there will be difference of accuracy in predicting the behaviour. either, might result far from the model. I assumed you knew this next statement, because you know how to use DOE. more levels in DOE, the more accurate you predict the physical behaviour of what you do. hard core experimenters love using minimum level 4 of DOE, multiple factors. Interpolation is more accurate than extrapolation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ pls do note, one of the ML calculation method to fit into the model, is also matrix multiplication. so No difference the way the ML (meant to physical experiment) and DOE (meant for physical experiment) calculate to fit into the model. $\endgroup$
    – Henry
    Nov 15, 2022 at 7:10
0
$\begingroup$

We have generally observed that nonlinear input-output dependencies captured by ML lead to better prediction than regression models developed using CCD DOE. We used ideantical data pairs for both. The RMS errors were generally lower for ML that carefully avoided overtraining. A good data collection strategy appears to be a grid design.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.