I have a question regarding support vector regression, best summarized by the chart below on simulated data of a linear function with a bit of noise. In essence, why does increasing epsilon rotate the fitted line away from the outlier?

Note: the simulated data is the red dots. Green line is a naive lm fit, blue lines are svr predictions for varying values of epsilon.

Motivation: trying to understand SVR in the face of a single large outlier.

enter image description here

Code here:

rm(list = ls())

n <- 1000
x <- 1:100
y <- 5*x + rnorm(length(x), 0, 10)
# y <- 500 #+ rnorm(length(x), 0, 10)
df <- data.frame(x, y)
df[nrow(df), ] <- c(100, 1000)  # add outlier at last x value
df <- df %>% arrange(x)

res <- list()  # blank list for results
epsilon <- seq(0,2.5,0.2)  # epsilon values to consider
# fit svr models
for (i in 1:length(epsilon)) {
  eps <- epsilon[i]
  svrModel <- svm(y ~ x, df, type = "eps-regression", kernel = "linear", epsilon = eps)
  svrTest <- predict(svrModel, df)
  res[[i]] <- data.frame(eps = eps, x = df$x, pred = svrTest)

# lm for comparison
lmModel <- lm(y ~ x, df)
predLM <- predict(lmModel, df)
dfLM <- data.frame(x = df$x, y = predLM)

resTidy <- bind_rows(res)  # tidy data

resTidy %>% 
  ggplot(aes(x = x, y = pred, group = eps, color = eps)) +
  geom_line(aes(colour = eps), size = 1.2) +
  geom_point(data = df, aes(x = x, y = y), alpha = 1, col = 'red') +
  geom_line(data = dfLM, aes(x = x, y = predLM), colour = 'green', linetype = 2, size = 1.2)

In particular the second link listed below (part way down) says, based on the Smola paper (link 3):

In SVR we want to maximize the prediction error in a defined precision, epsilon for better generalization. Here if we minimize the prediction error instead of maximize, the prediction result on unknown data is more likely to be over-fitted.

Thoughts greatly appreciated...finding SVR slightly unintuitive vs. SVM for classification.

P.S. Links that may be useful (I don't have enough reputation points to post all as hyperlinks) - remove the @@ which I inserted to be allowed to post them:

  1. https:@@//stats.stackexchange.com/questions/13194/support-vector-machines-and-regression
  2. https:@@//stats.stackexchange.com/questions/5945/understanding-svm-regression-objective-function-and-flatness
  3. https:@@//alex.smola.org/papers/2004/SmoSch04.pdf

This behavior is due to the fact that support vector regression fits the flattest possible function for a given epsilon-value. It is nicely illustrated in Fig. 3.28 of the lecture notes "Machine learning: Supervised techniques" by Sepp Hochreiter, currently here: http://www.bioinf.jku.at/teaching/current/ws_mlstvl/ML_supervised.pdf Your picture is nice though, in order to see what happens with outliers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.