Do most (all?) machine learning techniques share this architecture?

Question

Context: I've used Principal Component Analysis a fair bit, toyed with Artificial Neural Nets, and only read about other machine learning techniques. I have some academic exposure to machine learning, but only chapters of books and individual lectures - never a complete course or text.

---

There seem to be "architectural" similarities among machine learning techniques. There always seems to be a training process step and a prediction process step. Moreover, there seems to be similarities in the data passed among those two processes:

• Tagged or not, there is always training data.
• The training process always outputs a trained model
• The training process and/or model are usually configurable somehow.
• The prediction process uses the trained model to predict something about its input data.
• The training data and the prediction input data are both constrained in similar ways (EX: both are strings, or both are 30 dimensional vectors, or both are 16x16 images, etc).

Diagrammatically:

Do all machine learning techniques share this high level architecture?

Or, if this covers only a subset, what do people call that subset?

---

PCA and ANN, for example, both have a training step that takes training data to output a trained model. And both use that trained model to generate from data that hasn't yet been seen. Here's my attempt to hammer both of these techniques into the same diagram.

(Load the following images in a separate tab if you can't read them - they're designed for 150 pixels per inch.)

---

---

---

There seem to be a couple of established ways to do PCA (correlation method and covariance method), and neither one is very flexible or parametric. On the other hand, there seem to be N+1 interesting ways to configure or train neural nets, where N is the number of interesting ways in existence yesterday. Because of this, I imagine the difference gets pretty blurry between these two points:

• "two different ANN techniques"

vs

• "two different flavors/configurations of one ANN techique"

I'm assuming there are other ML techniques which are similar in that respect. I draw attention to this technique-vs-flavor ambiguity because it's the most complicating factor I've found when I'm trying to accurately shove a particular ML technique into the boxes of the diagram.

I also recognize that the training process can include the prediction process, as in neural nets or evolutionary techniques. But, as far as I know, there is still a distinct trained model that can be used with a prediction process after training.

Answer 1

I think that the process outlined in your upper diagram more accurately summarizes the subset called supervised learning. It's commonly presented in terms of function inference, the task of determining a (usually noisy) function from data. (See e.g. wikipedia.)

Though, it may not fully reflect all supervised methods. Take for example k nearest neighbors. When asked to classify a new instance, it queries the data for those nearest to the new one, and classifies or predicts, often using something like a vote or distance-weighted average. Importantly, it doesn't do any upfront learning to output a model.

More, PCA doesn't really fit this definition. It's a dimensionality-reduction technique that doesn't predict anything unknown about the data; it just projects it into a space of smaller dimension. If you use all of the principal components, you've simply expressed the data differently, and know nothing more or less about it than you would prior to using PCA. (Dimensionality reduction is many times categorized as unsupervised learning.)

For an idea of how one might define a common architecture, I'll share the definition that really helped me unify many topics in machine learning under a single umbrella. From Tom Mitchell in Machine Learning:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

For me, it's very intuitive to specify machine learning applications using this definition. For example:

• A program learns to predict housing prices (task) if the sum of squared error of its predictions (performance measure) decreases as its given more data (experience).
• A drone learns to efficiently order package deliveries and select routes if its speed improves as it delivers more of them.

The second example illustrates how this definition is broad enough to cover problems where training data doesn't come in the form of input-output tuples. I find it immensely useful for building a conceptual space that includes supervised learning, optimization and reinforcement learning. (I confess that it doesn't help me unify unsupervised learning, which I'm content to think of as either a precursor to some other learning task, or as an extension of descriptive statistics.)