# Which kernel function for Watson Nadaraya classifier?

I am trying to implement a Watson Nadaraya classifier. There is one thing I didn't understand from the equation:

$${F}(x)=\frac{\sum_{i=1}^n K_h(x-X_i) Y_i}{\sum_{i=1}^nK_h(x-X_i)}$$

What should I use for the kernel K?

I have a 2-dimensional dataset which has 1000 samples (each sample is like this: [-0.10984628, 5.53485135]).

What confuses me is, based on my data, the input of the kernel function will be something like this:

K([-0.62978309,  0.10464536])


And what I understand, it'll produce some number instead of an array, therefore I can go ahead and calculate F(x) which will also be a number. Then I'll check whether it is > or <= than zero. But I couldn't find any kernel that produces a number. So confused.

Edit: I tried to implement my classifier based on the comments, but I got a very low accuracy. I appreciate if someone notices what's wrong with it.

def gauss(x):
return (1.0 / np.sqrt(2 * np.pi)) * np.exp(- 0.5 * x**2)

def transform(X, h):
A = []
for i in X:
A.append(stats.norm.pdf(i,0,h)*stats.norm.pdf(i,0,h))
return A

N = 100
# pre-assign some mean and variance
mean1 = (0,9)
mean2 = (0,5)
cov = [[0.3,0.7],[0.7,0.3]]

# generate a dataset
dataset1 = np.random.multivariate_normal(mean1,cov,N)
dataset2 = np.random.multivariate_normal(mean2,cov,N)
X = np.vstack((dataset1, dataset2))

# pre-assign labels
Y1 = *N
Y2 = [-1]*N
Y = Y1 + Y2
# assing a width
h = 0.5

#now, transform the data
X2 = transform(X, h)

j = 0
predicted = []

for i in X2:
# apply the equation
fx = sum((gauss(i-X2))*Y)/float(np.sum(gauss(i-X2)))
# if fx>0, it belongs to class 1
if fx >0:
predicted.append(1)
else:
predicted.append(-1)
j = j+1

• Just to clarify (there seems to be some confusion in the answers below), you have a regression problem where you trying to predict y (a real number) given a multivariate x.... and you are interested in using a kernel regression approach, correct? – DavidR Oct 10 '11 at 18:26

You could take $K_h$ to be the density function for a bi-variate Gaussian distribution, with mean $x$, covariance matrix $hI$, and evaluated at $X_i$...

• Why Gaussian? Why is it appropriate for this problem? Does it have any optimal properties? – whuber Oct 10 '11 at 14:55
• I thought the question was more basic than 'what type of kernel function', and was more like 'how could this thing even work'... Nothing special about the Gaussian here, except computing it is an easy function in most platforms. – DavidR Oct 10 '11 at 18:33
• What about taking the norm of each row of the 2-D matrix (numpy.linalg.norm() in Python)? Does it sound reasonable? – user984041 Oct 11 '11 at 23:48
• I don't have time to look carefully at your code above right now, but a really good resource for these kernel regression methods is in the book Elements of Statistical Learning (www-stat.stanford.edu/~tibs/ElemStatLearn), which is available free, online. – DavidR Oct 12 '11 at 3:33

In your example the kernel function takes only one argument: $x-X_i$, so you definitely will not pass

K([-0.62978309,  0.10464536])


In general kernel always has another argument, $h$ -- the bandwith. You can use @DavidR advice, only for univariate case. Then your kernel function will be:

K <- function(x,h)dnorm(x,mu=0,sd=h)


So for your data the $F$ could be calculated in the following way:

F <- function(x,X,Y,h) {
kk <- K(x-X,h)
sum(kk*Y)/sum(kk)
}


You can find other examples in wikipedia. In this list the assumption is that $K_h(x)=K(x/h).$

Update

If $X$ is two-dimensional, then @DavidR advice applies directly:

K <- function(X,h) {
dnorm(X[,1],mu=0,sigma=h)*dnorm(X[,2],mu=0,sd=h)
}


here I used the fact that density of multivariate normal with zero mean and covariance matrix $hI$ is just a product of univariate normal densities. Note that I take sd=h, not sd=\sqrt{h} so that formula $K_h(x)=K(x/h)$ holds, so covariance matrix of the normal is actualy $h^2I$ in my code.

• I believe the OP uses "two-dimensional" to mean that $x - X_i$ is a 2-vector. They are asking for advice on what functional form $K$ should take. – whuber Oct 10 '11 at 14:55
• For the K([-0.62978309, 0.10464536]) example, I assumed I already subtracted x−Xi and divide it by the bandwidth. Because after I made these operations on a 2D sample, I will still have a 2D data like this: [-0.62978309, 0.10464536]. – user984041 Oct 10 '11 at 15:10
• hm, I assumed that 2 dimensions come from $(X,Y)$. If your data is 2D and $X$ has 2D, where does $Y$ come from? – mpiktas Oct 10 '11 at 18:04
• I am generating two different 2-D gaussian distribution, dataset1 = np.random.multivariate_normal(mean1,cov1,1000), dataset2 = np.random.multivariate_normal(mean2,cov2,1000), then I am combining them with: X = vstack((dataset1, dataset2)). Since I know which sample corresponds to which distribution, I have also Y. – user984041 Oct 10 '11 at 18:10
• I tried to apply this by doing: K=[] for i in X: K.append(stats.norm.pdf(i,0,h)*stats.norm.pdf(i,0,h)) but if I transform the data like this I get a very very low accuracy.. I found out stats.norm.pdf is the equalivent of dnorm in Python, but maybe I am using it wrong.. – user984041 Oct 11 '11 at 23:38