Our training data will either be linearly separable or non-linear separable. In both cases the decision boundary is defined by: $$ f(\mathbf{x}) = \mathbf{w} \cdot \mathbf{x} + b = 0 $$
and we classify them based on the value of $\sigma(\mathbf{w} \cdot \mathbf{x} + b)$. If this value is greater than 0.5 then the we classify as $y=1$ and and $y=0$ otherwise. That is we classify a point based on the side of the hyperplane it lies on.
When the data are linearly separable, then: $$\exists \mathbf{w},b \,\, \text{such that} \begin{cases} y = 1, \quad \text{for all training points with $y=1$} \\ y = 0, \quad \text{for all training points with $y=0$} \end{cases}$$
One can show that when the data are linearly separable the weights will go to infinity. In other words, gradient descent would not converge. Lets say that at the $k$-th iteration of the gradient descent we have the weights $\mathbf{w}_k$ and $b_k$ and these weights perfectly separate the data. Since $\mathbf{w}_k, b_k$ perfectly separate the data, so do the weights $\alpha \cdot \mathbf{w}_k, \alpha \cdot b_k$, where $\alpha > 0$.
So gradient descent would keep increasing the weights without bound. Is there a problem?
My reasoning is the following: lets say $\mathbf{w} = (w_1, w_2) = (2, 3)$ and $b=1$. The decision function is:
$$2x_1 + 3x_2 + 1 = 0$$
If these weights separate the data, so do the weights (20000, 30000) and $b=10000$ (just multiply the above equation with 10000). So the decision function doesn't change. The only thing that changes when weights get very large is the value returned by the sigmoid (see the figure). So if we want to classify a new point, the answered returned in both cases would be the same.
Now, in the answer of a related question it is stated that there is overfitting (in the case of separable data) when the weights go to infinity. Sorry, but I don't get it. When we regularize in Linear regression for example, we regularize in order to avoid fitting noise and not because large weights are somehow "bad". Although I understand, that penalizing the weights will prevent the model to achieve a zero training loss (predicted probabilities $\to 1$ for training points with $y=1$ and predicted probabilities $\to 0$ for training points with $y=0$), I can't understand how this prevents from overfitting.
Again from the figure, if $w=2$ was the weight we achieved when regularization was applied and $w=20$ when regularization was not applied, if we were asked to classify a point with $x=2$, we will predict $y=1$ in both cases, although with different levels of "confidence" (apologize if this term is not proper).
Can someone explain how large weights translate to overfitting when the data are linearly separable?
