My understanding of SVMs is the following:
The kernel trick allows us to project data from a training set which isn't linearly separable into a higher dimensional space where it becomes linearly separable. This in turn makes it possible to find the optimal separating boundary between the two classes, hence improving the generalization capabilities of the SVM compared to other methods such as (single hidden layer) perceptrons and k-NN.
In a discussion with someone over SVMs, the other person pointed out that increasing the number of dimensions of the data (i.e the kernel trick) would actually lead to more overfitting, since with enough dimensions, any data set becomes linearly and perfectly separable - including noisy data sets. This would lead to the generalization accuracy deteriorating instead of improving.
His reasoning makes sense, but then what purpose does the kernel trick serve? Why doesn't the kernel trick lead to more overfitting?