# SVR with combination of kernels

I'm trying to use Support Vector Regression (SVR) to model a trend.

I have a dataset with dependent variables $$Y$$ and features $$X$$. Suppose for simplicity that both $$Y$$ and $$X$$ take real values from $$-1$$ to $$1$$. When I plot $$Y$$ vs $$X$$, I get a non-trivial and definitely non-linear trend. I fit the dataset with the SVR scikit-learn regressor using an 'rbf' kernel function SVR(kernel = 'rbf').fit(X,Y). The result is not bad, but I want to improve it.

In particular, I notice that within the whole data set $$Y$$ vs $$X$$, there are some sub-trend. For example, for the range $$X_0 the trend is linear, for $$X_1 is quadratic etc. where $$X_0,X_1,X_2$$ are some values of $$X$$.

Now, I defined a different kernel for different range of $$X$$ values, and then used a linear combination of those kernels to model the whole dataset. In other words, something like this:

kernel1    = SVR(kernel = 'rbf').fit(X0_X1, Y0_Y1)
kernel2    = SVR(kernel = 'rbf').fit(X1_X2, Y1_Y2)
kernel_tot = SVR(kernel = a*kernel1 + b*kernel2 ).fit(X,Y)


where X0_X1 (X1_X2) represents the range of $$X$$ values between $$X_0$$ and $$X_1$$ ($$X_1$$ and $$X_2$$), and Y0_Y1 (Y1_Y2) the correspondending $$Y$$ values.

I expect that kernel1 will fit the first subset of data (X0_X1) well, the kernel2 the second subset (X1_X2), and the new defined kernel, a*kernel1 + b*kernel2, will fit the whole dataset with more accuracy than previously (i.e. by using one single rbf kernel for the whole dataset).

Is this a valid approach?

You are overcomplicating this. You can just filter your feature space and fit two separate models.

You can make new kernels but I can't find anything supporting your decision to use different kernels for different parts of your feature space.

• Hi, thanks for your answer. I see what you mean and I guess it makes sense. I could just use two models, one for each range. However, as I am learning now these things I was wondering if what I asked is possible, as I think is a more general solution. If nothing else, I am curious about it.
– Mash
Commented Oct 5, 2020 at 15:29