Support vector regression with one large outlier

Question

I have a question regarding support vector regression, best summarized by the chart below on simulated data of a linear function with a bit of noise. In essence, why does increasing epsilon rotate the fitted line away from the outlier?

Note: the simulated data is the red dots. Green line is a naive lm fit, blue lines are svr predictions for varying values of epsilon.

Motivation: trying to understand SVR in the face of a single large outlier.

Code here:

rm(list = ls())
library('e1071')
library('tidyverse')

n <- 1000
set.seed(250)
x <- 1:100
y <- 5*x + rnorm(length(x), 0, 10)
# y <- 500 #+ rnorm(length(x), 0, 10)
df <- data.frame(x, y)
df[nrow(df), ] <- c(100, 1000)  # add outlier at last x value
df <- df %>% arrange(x)

res <- list()  # blank list for results
epsilon <- seq(0,2.5,0.2)  # epsilon values to consider
# fit svr models
for (i in 1:length(epsilon)) {
  eps <- epsilon[i]
  svrModel <- svm(y ~ x, df, type = "eps-regression", kernel = "linear", epsilon = eps)
  svrTest <- predict(svrModel, df)
  res[[i]] <- data.frame(eps = eps, x = df$x, pred = svrTest)
}

# lm for comparison
lmModel <- lm(y ~ x, df)
predLM <- predict(lmModel, df)
dfLM <- data.frame(x = df$x, y = predLM)

resTidy <- bind_rows(res)  # tidy data

resTidy %>% 
  ggplot(aes(x = x, y = pred, group = eps, color = eps)) +
  geom_line(aes(colour = eps), size = 1.2) +
  geom_point(data = df, aes(x = x, y = y), alpha = 1, col = 'red') +
  geom_line(data = dfLM, aes(x = x, y = predLM), colour = 'green', linetype = 2, size = 1.2)

In particular the second link listed below (part way down) says, based on the Smola paper (link 3):

In SVR we want to maximize the prediction error in a defined precision, epsilon for better generalization. Here if we minimize the prediction error instead of maximize, the prediction result on unknown data is more likely to be over-fitted.

Thoughts greatly appreciated...finding SVR slightly unintuitive vs. SVM for classification.

P.S. Links that may be useful (I don't have enough reputation points to post all as hyperlinks) - remove the @@ which I inserted to be allowed to post them:

1. https:@@//stats.stackexchange.com/questions/13194/support-vector-machines-and-regression
2. https:@@//stats.stackexchange.com/questions/5945/understanding-svm-regression-objective-function-and-flatness
3. https:@@//alex.smola.org/papers/2004/SmoSch04.pdf

Answer 1

This behavior is due to the fact that support vector regression fits the flattest possible function for a given epsilon-value. It is nicely illustrated in Fig. 3.28 of the lecture notes "Machine learning: Supervised techniques" by Sepp Hochreiter, currently here: http://www.bioinf.jku.at/teaching/current/ws_mlstvl/ML_supervised.pdf Your picture is nice though, in order to see what happens with outliers.