I am confused with these machine learning terms, and trying to distinguish them with one concrete example.

for instance, use logistic regression to classify a bunch of cat images.

assume there are 1,000 images with labels indicating the corresponding image is or is not a cat image.

each image has a size of 100*100.

given above, is my following understanding right?

the sample space is the 1,000 images.

the feature space is 100*100 pixels.

the parameter space is a vector that has a length of 100*100+1.

the Hypothesis space is the set of all the possible hyperplanes that have some attribute that I have no idea.


People are a bit loose with their definitions (meaning different people will use different definitions, depending on the context), but let me put what I would say. I will do so more in the context of modern computer vision.

First, more generally, define $X$ as the space of the input data, and $Y$ as the output label space (some subset of the integers or equivalently one-hot vectors). A dataset is then $D=\{ d=(x,y)\in X\times Y \}$, where $d\sim P_{X\times Y}$ is sampled from some joint distribution over the input and output space.

Now, let $\mathcal{H}$ be a set of functions such that an element $f \in \mathcal{H}$ is a map $f: X\rightarrow Y$. This is the space of functions we will consider for our problem. And finally, let $g_\theta \in \mathcal{H}$ be some specific function with parameters $\theta\in\mathbb{R}^n$, such that we denote $\widehat{y} = g_\theta(x|\theta)$.

Finally, lets assume that any $f\in\mathcal{H}$ consists of a sequence of mappings $f=f_\ell\circ f_{\ell-1}\circ\ldots\circ f_2\circ f_1$, where $f_i: F_{i}\rightarrow F_{i+1}$ and $F_1 = X, \, F_{\ell+1}=Y$.

Ok, now for the definitions:

Hypothesis space (HS): the HS is the abstract function space you consider in solving your problem. Here it is denoted $\mathcal{H}$. I find that this term does not appear very often in applied ML, rather, it is mostly used in theoretical contexts (e.g., PAC theory).

Sample space (SS): the sample space is simply the input (or instance) space $X$. This is the same as in probability theory, regarding each training input as a random sample instance1.

Parameter space (PS): for a fixed classifier $g_\theta$, the PS is simply the space of possible values of $\theta$. It defines the space covered by the single architecture that you train2. Usually it does not include hyper-parameters when people say it.

Feature space (FS): for many models, there are multiple feature spaces. I've denoted them here as $F_2,\ldots, F_\ell$. They are essentially the intermediate outputs due to the model's layered processing (but see note1). For CNNs, these "feature maps" at different layers are often used for different things, hence distinction is important.

For your example:

  • The HS is almost the same as the PS once you've chosen logistic regression (except that the HS includes the models arising from different hyper-parameters as well, whereas the PS is fixed for a given set of hyper-parameters). Indeed, here, the HS is the set of all hyperplanes (and the PS could be as well, depending on the presence of e.g. regularization parameters).

  • The sample space is the set of all possible cat images; i.e., $X$. It is not usually restricted in meaning to be $D$, which is usually just called the training set.

  • The feature space in your case is indeed $F_1 = X$, assuming that you feed the raw pixels to the logistic regression (so $\ell = 1$).3

1Some people treat some processed form of the input as the input. E.g., replacing an image $I$ with its HOG or wavelet features $u(I)$. Then they define the sample space $X_u = \{ u(I_k) \;\forall\; k \}$, i.e., as the features rather than the images. However, I would argue that you should leave $I\in X$ and simply set $F_1 = X_u$, i.e., treat it as the first feature space.

2Note that each $\theta$ defines a different trained model, which is in the HS. However, not all members of $\mathcal{H}$ can be reached by varying the parameter vector. For instance, you might search over the number of layers in a CNN, but the parameter space of a single CNN will not cover that. (Though note again that $\mathcal{H}$ tends to be used more in theoretical contexts). One distinction between HS and PS appears in the context of error decompositions of approximation vs estimation noise.

3Normally (in "older" computer vision) you would extract features from the image and feed that to e.g. logistic regression. The modern version of this is attaching a fully connected (linear) layer with a softmax at the end of a CNN.


I'll approach this from a more colloquial point of view:

The sample space consists of your sample-level input data, which are instances of specific values in feature space. In your example, your sample space consists of 1000 images.

The feature space consists of the individual components that make up a sample, and potentially intermediate, derived features that express combinations of the raw features. In your example, the feature space is the 10,000 pixels and the color values they can take.

The hypothesis space covers all potential solutions that you could arrive at with your choice of model. A model that draws a linear boundary in feature space, for example, does not have any nonlinear solutions in its hypothesis space. In most cases, you can't enumerate the hypothesis space, but it's useful to know what types of solutions it's even possible for your model to generate.

The parameter space covers the possible values that the model parameters can take, which will vary depending on your model. A logistic regression, for example, will have a weight parameter for every feature that varies between -Inf and +Inf. You could also build a coin flip model that guesses "cat" randomly with probability X, where X is the single parameter that varies from 0 to 100.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.