2
$\begingroup$

I'm trying to use Support Vector Regression (SVR) to model a trend.

I have a dataset with dependent variables $Y$ and features $X$. Suppose for simplicity that both $Y$ and $X$ take real values from $-1$ to $1$. When I plot $Y$ vs $X$, I get a non-trivial and definitely non-linear trend. I fit the dataset with the SVR scikit-learn regressor using an 'rbf' kernel function SVR(kernel = 'rbf').fit(X,Y). The result is not bad, but I want to improve it.

In particular, I notice that within the whole data set $Y$ vs $X$, there are some sub-trend. For example, for the range $X_0<X<X_1$ the trend is linear, for $X_1<X<X_2$ is quadratic etc. where $X_0,X_1,X_2$ are some values of $X$.

Now, I defined a different kernel for different range of $X$ values, and then used a linear combination of those kernels to model the whole dataset. In other words, something like this:

kernel1    = SVR(kernel = 'rbf').fit(X0_X1, Y0_Y1) 
kernel2    = SVR(kernel = 'rbf').fit(X1_X2, Y1_Y2) 
kernel_tot = SVR(kernel = a*kernel1 + b*kernel2 ).fit(X,Y) 

where X0_X1 (X1_X2) represents the range of $X$ values between $X_0$ and $X_1$ ($X_1$ and $X_2$), and Y0_Y1 (Y1_Y2) the correspondending $Y$ values.

I expect that kernel1 will fit the first subset of data (X0_X1) well, the kernel2 the second subset (X1_X2), and the new defined kernel, a*kernel1 + b*kernel2, will fit the whole dataset with more accuracy than previously (i.e. by using one single rbf kernel for the whole dataset).

Is this a valid approach?

$\endgroup$

1 Answer 1

1
$\begingroup$

You are overcomplicating this. You can just filter your feature space and fit two separate models.

You can make new kernels but I can't find anything supporting your decision to use different kernels for different parts of your feature space.

$\endgroup$
1
  • $\begingroup$ Hi, thanks for your answer. I see what you mean and I guess it makes sense. I could just use two models, one for each range. However, as I am learning now these things I was wondering if what I asked is possible, as I think is a more general solution. If nothing else, I am curious about it. $\endgroup$
    – Mash
    Commented Oct 5, 2020 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.