I try to understand what exactly we are trying to optimize in the case of Support Vector Machine problem, which supports soft margins. The original problem is posed first as, without soft margins (assuming linear separability):
\begin{equation*} \begin{aligned} & \underset{\gamma}{\text{maximize}} & & \dfrac{\gamma}{||w||} \\ & \text{subject to} & & y^{(i)}(w^Tx^{(i)} +b) \geq \gamma, \; i = 1, \ldots, m. \end{aligned} \end{equation*}
$\gamma$ is the so called functional margin of the closest sample to the hyperplane, related to the actual geometric margin $\gamma' = \dfrac{\gamma}{||w||}$. Then in order to make this a convex problem, the closest functional margin is equated to $1$. Given a hyperplane $(w,b)$ where $w$ is a unit vector; all $(\alpha w,\alpha b)$ pairs are equivalent where $\alpha > 0$. By equating $\gamma$ to $1$, we favor the $(\alpha' w,\alpha' b)$ pair where $\alpha' = \dfrac {1}{\gamma'}$ over all $(\alpha w,\alpha b)$ pairs. The objective turns to $\dfrac{1}{||w||}$ whose minimization is equivalent to the maximization of $||w||^2$. The final problem is then:
\begin{equation*} \begin{aligned} & \underset{w}{\text{minimize}} & & \ ||w||^2 \\ & \text{subject to} & & y^{(i)}(w^Tx^{(i)} +b) \geq 1, \; i = 1, \ldots, m. \end{aligned} \end{equation*}
If we allow points which violate the margin conditions, by penalizing them, we obtain the soft margin support vector machine problem, which is then: \begin{equation*} \begin{aligned} & \underset{w}{\text{minimize}} & & \ ||w||^2 + C \sum_{1}^{m} \epsilon_i\\ & \text{subject to} & & y^{(i)}(w^Tx^{(i)} +b) \geq 1 - \epsilon_i, \; i = 1, \ldots, m. & & \epsilon_i \geq 0, \; i = 1, \ldots, m. \end{aligned} \end{equation*}
Here is something which I not clearly understand: How the penalty values, $\epsilon_i$, should be interpreted? We again consider the smallest positive functional margin to be equal to $1$ such that $\alpha' = \dfrac{1}{\gamma'}$ where is $\gamma'$ is the smallest positive geometric margin with respect to the hyperplane. Then the penalty $\epsilon_i$ which we pay to shift a data point to its closest admissible location becomes dependent on $\gamma'$. For example if we pick a hyperplane where $\gamma'$ is large, then $\alpha'$ becomes small and for the same amount of actual geometric distance, $\epsilon_i$ becomes smaller to compared a hyperplane where $\gamma'$ is small. In the end, different hyperplanes pay different costs for moving inadmissible data points at the same distances; hyperplanes with large nearest geometric margin are favoured.
How is this behavior justified? Shouldn't we design a method which assigns the same cost for the same distance among all hyperplanes?