5
$\begingroup$

I am reading Bishop - Pattern Recognition and Machine Learning book. In Chapter 4 first page, there is a statement that

In this chapter, we consider linear models for classification, by which we mean that the decision surfaces are linear functions of the input vector x and hence are defined by (D − 1)-dimensional hyper planes within the D-dimensional input space.

If the input vector is of D dimensions then the decision boundary should also be of D dimensions. Why is it D-1? For example, in 2-D plane, the line separating the points is also a straight line which is of 2 dimensions.

Am I thinking something terribly wrong? Can anyone explain me how is it D-1 dimensions for decision boundary?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Not so; the line is considered 1 dimensional. $\endgroup$
    – Nick Cox
    Commented Jan 8, 2018 at 13:49
  • 1
    $\begingroup$ There is a unique $D$-dimensional linear (or affine) subspace of any (finite) $D$-dimensional space: the space itself. That would be useless as a decision boundary, wouldn't it? $\endgroup$
    – whuber ♦
    Commented Jan 8, 2018 at 14:02

2 Answers 2

Reset to default
10
$\begingroup$

The line is a 1-D boundary in 2-D space. If you think of yourself as a point on the decision boundary, the number of (non-parallel nor anti-parallel) directions you could travel on the boundary will be its dimensions. With a line you can go forward or backward (which is anti-parallel to forward), so there is only one dimension. A 2-D plane would let you go forward-backwards or left-right which makes it 2-D.

In the images below the decision boundaries for 2D and 3D input space are shown in blue. The orange lines illustrate that these decision boundaries have D$-1$ dimensions themselves.

enter image description here

Think about rotating the image. You can rotate it so that the blue line becomes horizontal (i.e. super clearly 1D) but no matter how you rotate it the plane created by the black axes will still be 2D as seen in this image. This is because the blue line is a 1D slice in 2D space:

enter image description here

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I agree that a straight line is a 1D object and a plane is a 2D object. But if the input space has two co-ordinates or features then we represent the straight line or decision boundary as ax+by+c=0 i.e., by using two dimensions. Is the author in this context discussing only about their physical appearance or their representation in vector form? $\endgroup$
    – bharadwaj
    Commented Jan 9, 2018 at 15:13
  • 2
    $\begingroup$ @bharadwaj $ax+by+c=0$ is because it exists in 2D space, but it is itself a 1D slice of that 2D space. In my images, the number of black lines (i.e. the axes) show the dimensionality of the space but the orange lines show the dimensionality of the boundary and it is always one less than that of the space. $\endgroup$
    – Dan
    Commented Jan 9, 2018 at 15:26
  • $\begingroup$ @bharadwaj I've added an extra explanation, does that make it any clearer? $\endgroup$
    – Dan
    Commented Jan 9, 2018 at 15:32
0
$\begingroup$

I think the answer of @Dan is sufficient, but for someone who have basic Linear Algebra, this is another explanation. It is obvious the discriminant function f is a linear transformation, and vector in the boundary decision form the null-space. It is known that dim(V) = dim(null-space) + dim(f(V)), with V is the input space. Since dim(f(V)) = 1, dim(null-space) = D - 1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.