Can someone please explain to me what the particular scenarios mean?

Question

"The set of points in $\mathbb{R}^2$ classified ORANGE corresponds to {$x:x^Tβ>0.5$}, indicated in Figure 2.1, and the two predicted classes are separated by the decision boundary {$x:x^Tβ=0.5$}, which is linear in this case. We see that for these data there are several misclassifications on both sides of the decision boundary. Perhaps our linear model is too rigid—or are such errors unavoidable? Remember that these are errors on the training data itself, and we have not said where the constructed data came from. Consider the two possible scenarios:

Scenario 1: The training data in each class were generated from bivariate Gaussian distributions with uncorrelated components and different means.

Scenario 2: The training data in each class came from a mixture of 10 low- variance Gaussian distributions, with individual means themselves distributed as Gaussian.

A mixture of Gaussians is best described in terms of the generative model. One first generates a discrete variable that determines which of the component Gaussians to use, and then generates an observation from the chosen density. In the case of one Gaussian per class, we will see in Chapter 4 that a linear decision boundary is the best one can do, and that our estimate is almost optimal. The region of overlap is inevitable, and future data to be predicted will be plagued by this overlap as well. In the case of mixtures of tightly clustered Gaussians the story is different. A linear decision boundary is unlikely to be optimal, and in fact is not. The optimal decision boundary is nonlinear and disjoint, and as such will be much more difficult to obtain."

Can someone please explain to me what the particular scenarios mean? It's from Elements of Statistical Learning by Tibshirani

Answer 1

In scenario 1, there are two bivariate Normal distributions. Here I show two such probability density functions (PDFs) superimposed in a pseudo-3D plot. One has a mean near $(0,0)$ (at the left) and the other has a mean near $(3,3)$.

Samples are drawn independently from each. I took the same number ($300$) so that we wouldn't have to compensate for different sample sizes in evaluating these data.

Point symbols distinguish the two samples. The gray/white background is the best discriminator: points in gray are more likely to arise from the second distribution than the first. (The discriminator is elliptical, not linear, because these distributions have slightly different covariance matrices.)

In scenario 2 we will look at two comparable datasets produced using mixture distributions. There are two mixtures. Each one is determined by ten distinct Normal distributions. They all have different covariance matrices (which I do not show) and different means. Here are the locations of their means (which I have termed "nuclei"):

A mixture of Gaussians is best described in terms of the generative model. One first generates a discrete variable that determines which of the component Gaussians to use, and then generates an observation from the chosen density.

To draw an set of independent observations from a mixture, you first pick one of its components at random and then draw a value from that component. The PDF of a mixture is a weighted sum of PDFs of the components, with the weights being the chance of selecting each component in that first stage. Here are the PDFs of the two mixtures. I drew them with a little extra transparency so you can see them better in the middle where they overlap:

To make the two scenarios easier to compare, the means and covariance matrices of these two PDFs we chosen to closely match the corresponding means and covariances of the two bivariate Normal PDFs used in scenario 1.

To emulate scenario 2 (the mixture distributions), I drew samples of 300 independent values from each of the two datasets by selecting each of their components with a probability of $1/10$ and then independently drawing a value from the selected component. Because the selection of components is random, the number of draws from each component was not always exactly $30 = 300 \times 1/10$, but it was usually close to that. Here is the result:

The black dots show the ten component means for each of the two distributions. Clustered around each black dot are approximately 30 samples. However, there is much intermingling of values, so it is impossible from this figure to determine which samples were drawn from which component.

In the case of mixtures of tightly clustered Gaussians the story is different. A linear decision boundary is unlikely to be optimal, and in fact is not. The optimal decision boundary is nonlinear and disjoint, and as such will be much more difficult to obtain."

The background in that last figure is the best discriminator for these two mixture distributions. It is complicated because the distributions are complicated; obviously it is not just a line or smooth curve, such as appeared in scenario 1.

I believe the entire point of this comparison lies in our option, as analysts, to choose which model we want to use to analyze either one of these two datasets. Because we would not in practice know which model is appropriate, we could try using a mixture model for the data in scenario 1 and we could equally well try using a Normal model for the data in scenario 2. We would likely be fairly successful in any case due to the relatively low overlap (between blue and red sample points). Nevertheless, the different (equally valid) models can produce distinctly different discriminators (especially in areas where data are sparse).

Answer 2

The point being made in section 2.3 of the book (where this quote come from) is that if the source of the data is from Scenario 1, there is nothing better you can do than a linear division (as in figure 2.1). Any finer tuning is actually self-delusion: you should then expect to get worse results predicting cases outside the training data if you do not use the optimal linear division.

However, if the source of the data is from Scenario 2, you can reasonably expect the low variance of each of the $10$ source distributions to make it more likely that data points of the same colour will tend to cluster together in a non-linear manner, and so a non-linear approach may be more skilful. The example the book gives is that of looking at the colours of nearest neighbours: figure 2.2 shows the classification boundary if you look at the 15 nearest neighbours (a fairly smooth non-linear boundary) while figure 2.3 looks at the boundary if you look at just 1 nearest neighbour (a very jagged boundary).

I suspect that the point being made is that the value of statistical or machine learning techniques depends on the source of the data, and that some techniques are better in some circumstances, and others in others. But it is also possible to generalise ideas from different methods and come up with further techniques, as section 2.4 and figure 2.5 do with what the book calls the "Bayes classifier".

Answer 3

whuber made an extraordinary good statement. Here I just want to add some more details.

1. The scenario 1 is talking about linear discriminative analysis(LDA) where the decision boundary is linear and whuber is describing a more general quadratic discriminative analysis(QDA) where the decision boundary is a quadratic function. Of course, LDA is a special case of QDA.

2. A linear decision boundary is optimal to scenario 1 because solving the classification problem using maximum likelihood estimation gives a linear solution.

3. At the same time, even though LDA looks very different from linear regression, the decision boundaries given by these two methods are very similar. Intuitively, if we think these two decision boundaries are two straight lines, these two lines will have the same slope but different intercepts.

For more mathematical details, I would recommend this blog which gives a great and detailed explanation.