Understanding SVM regression: objective function and "flatness" SVMs for classification make intuitive sense to me: I understand how minimizing $||\theta||^2$ yields the maximum margin. However, I don't understand that objective in the context of regression. Various texts (here and here) describe this as maximizing "flatness." Why would we want to do that? What in regression is the equivalent to the concept of "margin"?
Here are a few attempted answers, but none that really helped my understanding.
 A: shabbychef gave a very clear explanation from the perspective of the model complexity. I will try to understand this problem from another point of view in case it may help anyone. 
Basically we want to maximize the margin in SVC. This is the same in SVR while we want to maximize the prediction error in a defined precision $e$ for better generalization. Here if we minimize the prediction error instead of maximize, the prediction result on unknown data is more likely to be overfitted. Let us think about the "maximize the prediction error" in one-dimensional case. 
In one-dimensional case, our goal is to maximize the distances from all points $(x_i,y_i)$ to the trend line $y=\omega x+b$ within $e$. Note that we set the constrain of precision as $e$ so that we can maximize the distance, not minimize. Then let us take a look at the very simple equation of the distance from a point to a line.
$$
\frac{\left|\omega x_i-y_i+b\right|}{\sqrt {\omega^2+1}}
$$
Right now the numerator is limited to $e$. To maximize the distance, what we try to do is to minimize $\omega$.
Anyone can easily extend the one-dimensional case to N-dimensional case as the distance equation will always be Euclidean distance.
Additionally, we may have a review on the optimization problem in SVR for the comparison [1].
$$
\min \frac{1}{2} {\left| \left| \omega \right| \right|}^2
$$
$$
s.t. \begin{cases}y_i-<\omega,x_i>-b \leq e\\<\omega,x_i>+b-y_i \geq e\end{cases}
$$
Thanks.
[1] Smola, A., and B. Schölkopf. A tutorial on support vector regression. Statistics and Computing, Vol. 14, No. 3, Aug. 2004, pp. 199–222.
A: One way that I think about the flatness is that it makes my predictions less sensitive to perturbations in the features. That is, if I am constructing a model of the form
$$y = x^\top \theta + \epsilon,$$
where my feature vector $x$ has already been normalized, then smaller values in $\theta$ mean my model is less sensitive to errors in measurement/random shocks/non-stationarity of the features, $x$. Given two models (i.e. two possible values of $\theta$) which explain the data equally well, I prefer the 'flatter' one. 
You can also think of Ridge Regression as peforming the same thing without the kernel trick or the SVM 'tube' regression formulation.
edit: In response to @Yang's comments, some more explanation:


*

*Consider the linear case: $y = x^\top \theta + \epsilon$. Suppose the $x$ are drawn i.i.d. from some distribution, independent of $\theta$. By the dot product identity, we have $y = ||x|| ||\theta|| \cos\psi + \epsilon$, where $\psi$ is the angle between $\theta$ and $x$, which is probably distributed under some spherically uniform distribution. Now note: the 'spread' (e.g. the sample standard deviation) of our predictions of $y$ is proportional to $||\theta||$. To get good MSE with the latent, noiseless versions of our observations, we want to shrink that $||\theta||$. c.f. James Stein estimator.

*Consider the linear case with lots of features. Consider the models $y = x^\top \theta_1 + \epsilon$, and $y = x^\top \theta_2 + \epsilon$. If $\theta_1$ has more zero elements in it than $\theta_2$, but about the same explanatory power, we would prefer it, base on Occam's razor, since it has dependencies on fewer variables (i.e. we have 'done feature selection' by setting some elements of $\theta_1$ to zero). Flatness is kind of a continuous version of this argument. If each marginal of $x$ has unit standard deviation, and $\theta_1$ has e.g. 2 elements which are 10, and the remaining $n-2$ are smaller than 0.0001, depending on your tolerance of noise, this is effectively 'selecting' the two features, and zeroing out the remaining ones.

*When the kernel trick is employed, you are performing a linear regression in an high (sometimes infinite) dimensional vector space. Each element of $\theta$ now corresponds to one of your samples, not your features. If $k$ elements of $\theta$ are non-zero, and the remaining $m-k$ are zero, the features corresponding to the $k$ non-zero elements of $\theta$ are called your 'support vectors'. To store your SVM model, say on disk, you need only keep those $k$ feature vectors, and you can throw the rest of them away. Now flatness really matters, because having $k$ small reduces storage and transmission, etc, requirements. Again, depending on your tolerance for noise, you can probably zero out all elements of $\theta$ but the $l$ largest, for some $l$, after performing an SVM regression. Flatness here is equivalent to parsimony with respect to the number of support vectors.

A: At least, I don't think minimizing $\theta$ has anything to do with the concept margin as in a SVM classification setting. 
It serves for a totally different goal that is well explained by the above two posts, i.e., reducing model complexity and avoiding overfitting.
