K-nearest-neighbour with continuous and binary variables

Question

I have a data set with columns a b c (3 attributes). a is numerical and continuous while band c are categorical each with two levels. I am using the K-Nearest Neighbors method to classify aand b on c. So, to be able to measure the distances I transform my data set by removing b and adding b.level1and b.level2. If observation i has the first level in the bcategories, b.level1[i]=1 and b.level2[i]=0.

Now I can measure distances in my new data set: a b.level1 b.level2

From a theoretical/mathematical point of view: Can you perform K-nearest neighbor (KNN) with both binary and continuous data?

I am using FNNpackage in R and the function knn()

Answer 1

It's ok combining categorical and continuous variables (features).

Somehow, there is not much theoretical ground for a method such as k-NN. The heuristic is that if two points are close to each-other (according to some distance), then they have something in common in terms of output. Maybe yes, maybe no. And it depends on the distance you use.

In your example you define a distance between two points $(a,b,c)$ and $(a',b',c')$ such as :

• take the squared distance between $a$ and $a'$ : $(a-a')^2$
• Add +2 if $b$ and $b'$ are different, +0 if equal (because you count a difference of 1 for each category)
• Add +2 if $c$ and $c'$ are different, +0 is equal (same)

This corresponds to giving weights implicitly to each feature.

Note that if $a$ takes large values (like 1000, 2000...) with big variance then the weights of binary features will be negligible compared to the weight of $a$. Only the distance between $a$ and $a'$ will really matter. And the other way around : if $a$ takes small values like 0.001 : only binary features will count.

You may normalize the behaviour by reweighing: dividing each feature by its standard deviation. This applies both to continuous and binary variables. You may also provide your own preferred weights.

Note that R function kNN() does it for you : https://www.rdocumentation.org/packages/DMwR/versions/0.4.1/topics/kNN

As a first attempt, just use basically norm=true (normalization). This will avoid most non-sense that may appear when combining continuous and categorical features.

Answer 2

Yes, you certainly can use KNN with both binary and continuous data, but there are some important considerations you should be aware of when doing so.

The results are going to be heavily informed by the binary splits relative to the dispersion among the real-valued results (for 0-1 scaled, unweighted vectors), as illustrated below:

You can see in this example that an individual observation's nearest neighbors by distance would be MUCH more heavily informed by the binary variable than by the scaled real-value variable.

Furthermore, this extends to multiple binary variables- if we change one of the real-valued variables to binary, we can see that the distances will by much more informed by matching on all of the binary variables involved than in nearness of the real values:

You'll want to include only critical binary variables- you are, in effect, asking "of all of the observations that match this configuration of binary variables (if any), which have the nearest real-valued values?" This is a reasonable formulation of many problems that could be addressed with KNN, and a very poor formulation of other problems.

#code to reproduce plots:
library(scatterplot3d) 

scalevector <- function(x){(x-min(x))/(max(x)-min(x))}

x <- scalevector(rnorm(100))
y <- scalevector(rnorm(100))
z <- ifelse(sign(rnorm(100))==-1, 0, 1)
df <- data.frame(cbind(x,y,z))

scatterplot3d(df$x, df$z, df$y, pch=16, highlight.3d=FALSE,
              type="h", angle =235, xlab='', ylab='', zlab='')

x <- scalevector(rnorm(100))
y <- ifelse(sign(rnorm(100))==-1, 0, 1)
z <- ifelse(sign(rnorm(100))==-1, 0, 1)
df <- data.frame(cbind(x,y,z))

scatterplot3d(df$x, df$z, df$y, pch=16, highlight.3d=FALSE,
              type="h", angle =235, xlab='', ylab='', zlab='')