I have a question regarding support vector regression, best summarized by the chart below on simulated data of a linear function with a bit of noise. In essence, why does increasing epsilon rotate the fitted line away from the outlier?
Note: the simulated data is the red dots. Green line is a naive lm fit, blue lines are svr predictions for varying values of epsilon.
Motivation: trying to understand SVR in the face of a single large outlier.
Code here:
rm(list = ls())
library('e1071')
library('tidyverse')
n <- 1000
set.seed(250)
x <- 1:100
y <- 5*x + rnorm(length(x), 0, 10)
# y <- 500 #+ rnorm(length(x), 0, 10)
df <- data.frame(x, y)
df[nrow(df), ] <- c(100, 1000) # add outlier at last x value
df <- df %>% arrange(x)
res <- list() # blank list for results
epsilon <- seq(0,2.5,0.2) # epsilon values to consider
# fit svr models
for (i in 1:length(epsilon)) {
eps <- epsilon[i]
svrModel <- svm(y ~ x, df, type = "eps-regression", kernel = "linear", epsilon = eps)
svrTest <- predict(svrModel, df)
res[[i]] <- data.frame(eps = eps, x = df$x, pred = svrTest)
}
# lm for comparison
lmModel <- lm(y ~ x, df)
predLM <- predict(lmModel, df)
dfLM <- data.frame(x = df$x, y = predLM)
resTidy <- bind_rows(res) # tidy data
resTidy %>%
ggplot(aes(x = x, y = pred, group = eps, color = eps)) +
geom_line(aes(colour = eps), size = 1.2) +
geom_point(data = df, aes(x = x, y = y), alpha = 1, col = 'red') +
geom_line(data = dfLM, aes(x = x, y = predLM), colour = 'green', linetype = 2, size = 1.2)
In particular the second link listed below (part way down) says, based on the Smola paper (link 3):
In SVR we want to maximize the prediction error in a defined precision, epsilon for better generalization. Here if we minimize the prediction error instead of maximize, the prediction result on unknown data is more likely to be over-fitted.
Thoughts greatly appreciated...finding SVR slightly unintuitive vs. SVM for classification.
P.S. Links that may be useful (I don't have enough reputation points to post all as hyperlinks) - remove the @@ which I inserted to be allowed to post them:
- https:@@//stats.stackexchange.com/questions/13194/support-vector-machines-and-regression
- https:@@//stats.stackexchange.com/questions/5945/understanding-svm-regression-objective-function-and-flatness
- https:@@//alex.smola.org/papers/2004/SmoSch04.pdf