0
$\begingroup$

I am trying to perform multi-class classification using SVMs (C-SVC). I am using the ksvm function from the kernlab package in R. The problem is that I want to use a periodic kernel, which is not one of the kernel functions provided by kernlab. Reading through the documentation and based on a post I found here, I saw that one can compute the kernel matrix on the input data with a user-defined kernel using the kernelMatrix function and then input the resulting matrix in ksvm. However, the results do not really seem to agree for kernels that have already been defined. As a reference, I am including some R code used to replicate the laplacedot kernel results. Note that dtframe is a dataframe with 2880 observations, 2 continuous and one discrete variable and I am trying to predict the discrete variable (this is feature number 1 and takes 4 levels) using the 2nd and the 3rd variables (both continuous).

# Data generation
library(mvtnorm)
library(clusterGeneration)
library(kernlab)

# categorized numerical variable function
intv <- function(vec, class) {
  nbase <- (1:(class-1))/class
  nq <- numeric(length(nbase))
  for (i in 1:length(nq)) {
    nq[i] <- quantile(vec, nbase[i])
  }
  res <- c(min(vec), nq, max(vec)) 
  res[1] <- res[1]-1
  for (i in 2:length(res)){
    if (res[i-1]==res[i]){
      res[i] <- res[i]+2e-15
    }
  }
  return(res)
}

nobs <- 2880
nvars <- 3
set.seed(1234)
# Generate covariance matrix
sigma_mat <- genPositiveDefMat(dim = nvars,
                               covMethod = "unifcorrmat",
                               alphad = 3,
                               rangeVar = c(0.1, 5))

dtframe <- rmvnorm(nobs, mean=rep(0, nvars), sigma = sigma_mat$Sigma)

# Discretise first variable
dtframe[,1] <- as.factor(cut(dtframe[,1], intv(dtframe[,1], 4), labels = (1:4)))

# Create concentric circles
j <- 1
n_lvls <- length(unique(dtframe[,j]))
qs <- c(min(dtframe[, 2]^2+dtframe[, 3]^2))
for (i in 1:n_lvls){
  qs <- c(qs, quantile(dtframe[, 2]^2+dtframe[, 3]^2, i/n_lvls))
}
for (i in 1:nrow(dtframe)){
  val <- dtframe[i, 2]^2+dtframe[i, 3]^2
  # Find position where val lies
  new_lvl <- which(diff(sign(qs-val))!=0)
  if (length(new_lvl)>1){
    new_lvl <- sample(new_lvl, 1)
  }
  dtframe[i, j] <- new_lvl
}

# Manual implementation

laplacekern <- function(x, y=NULL, sigma = 1){
  return(exp(-sigma * sqrt(-(round(2 * crossprod(x, y) - crossprod(x) - crossprod(y), 9)))))
}

class(laplacekern) <- "kernel"

K <- kernelMatrix(laplacekern, as.matrix(dtframe[, c(2:3)]))

kernsvm <- ksvm(y = as.factor(dtframe[, 1]),
                x = K,
                kernel = 'matrix',
                type = 'C-svc',
                prob.model = TRUE)
svmpredictions <- predict(kernsvm, K)
cat('Misclassified observations:', sum(svmpredictions!=dtframe[, 1]), '\n')
"Misclassified observations: 2070"

# Using the built-in laplacedot

kernsvm_2 <- ksvm(y = as.factor(dtframe[, 1]),
                  x = as.matrix(dtframe[, c(2:3)]),
                  kernel = 'laplacedot',
                  params = list('sigma'=1),
                  type = 'C-svc',
                  prob.model = TRUE)
svmpredictions_2 <- predict(kernsvm_2, as.matrix(dtframe[, c(2:3)]))
cat('Misclassified observations:', sum(svmpredictions_2!=dtframe[, 1]), '\n')
"Misclassified observations: 66"

There is a huge difference between the results, which probably shows that there is something wrong with my attempt to perform multi-class classification using the kernel matrix instead of the kernel function (I know that the Laplace kernel should work for this data by the way, as I have generated them myself using the code above; I further include a plot of the data below). The code for the laplacekern function is taken directly from the source code of laplacedot. I am mainly wondering what I'm doing wrong in my manual implementation. Any help will be very much appreciated - thanks!

Plot of predictors - the colours represent the target variable levels.

$\endgroup$
2
  • $\begingroup$ Your code is not reproducible. A reproducible example is more likely to be answered $\endgroup$
    – Firebug
    Feb 3, 2023 at 12:43
  • $\begingroup$ Updated, so that the data generation is included, as well as a plot of what the data looks like. $\endgroup$
    – HeyCool08
    Feb 3, 2023 at 13:44

1 Answer 1

0
$\begingroup$

The scaled argument documentation states that:

scaled: A logical vector indicating the variables to be scaled. If scaled is of length 1, the value is recycled as many times as needed and all non-binary variables are scaled. Per default, data are scaled internally (both x and y variables) to zero mean and unit variance. The center and scale values are returned and used for later predictions.

One possibility is that you are not scaling your variables before computing the kernel matrix

$\endgroup$
1
  • $\begingroup$ Thanks for this - I tried it out but it still seems to be giving the same issues so I would imagine it doesn't have to do with scaling only. $\endgroup$
    – HeyCool08
    Feb 3, 2023 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.