SVM training yields too many (or no) support vectors

So I implemented a support vector machine, using either a linear kernel or the rbf-kernel. I trained and tested it on a two dimensional set of data and everything seems to be working fine.

However, then I tried training it on high dimensional data with few samples: Precisely speaking, I am using the complete images from the MNIST-data set, where the labels are $$0$$ or $$1$$.

I then measured performance by cross validtion (I train the SVM on 65 images, then evaluate the performance on the rest of the set) and realised the following issues:

1. My optimal quadratic programming solution $$\alpha$$ is very small. That way, it is hard to decide what entry is a support vector and what is not. Every entry usually is smaller than $$10^{-6}$$.
2. In the linear case detecting support vectors when $$\alpha_i$$ is greater than $$10^{-7}$$ works. I get on average 5 support vectors and an average test error of $$0.00316$$ (using the hard margin, no $$C$$ parameter). However, when introducing the soft margin by the regularization parameter $$C$$ and using the rbf-kernel the SVM behaves way worse. On average, the support vectors are the whole dataset and the train as well as the test error are at approximately $$50\%$$. I have tried changing around $$C$$ or $$\sigma$$, but the error rate does not change much.

I am sure with the rbf kernel the performance should be way better than $$50\%$$. I have also read that a high amount of support vectors could mean overfitting, but that does not seem to be the case here. Any help would be greatly appreciated!

Edit: I just compared with sklearn's SVM class but seem to get a similarly high error. I do not understand why rbf would perform so badly in this case compared to a linear kernel.