I know that Support Vector Machines are very well suited for high-dimensional data, and I have read that one reason is that they have an "inbuilt feature selection". I assume this is due to the fact that Support Vectors are used to build the decision boundary, and other weights could be regularized, i.e. set close to zero. Is this correct, and if so does anyone have this formally or helpful sources?
I think you are on the right track. The essence of it is that an SVM adds dimensions and places the data in that new space - possibly infinite in number of dimensions - and looks for the hyperplane that best separates. In practice, some vectors get lightly weighted. I view it as 'inbuilt' in that - unlike a neural net - it doesn't have to hunt hoping to find a solution; there either is such a hyperplane or there isn't.
However, how and what happens depends a great deal on the kernel you choose and how it is tuned.
A great lecture is here: https://www.youtube.com/watch?v=eHsErlPJWUU