# Why do model weights become less explainable with more training?

For example, I was using a logistic regression on the Fashion-MNIST dataset. This is using sklearn, which uses an iterative training approach, and I was experimenting with the number of iterations. (This is with no regularisation, and using the "multinomial" approach to multiclass, though one-versus-rest yields the same thing.) Here are what the coefficients look like for each class, for different numbers of training iterations:

The range of coefficients is increasing with number of iterations:

And here are the training and validation accuracies at each number of iterations:

What surprises me is that the coefficients become less explainable and intuitive with more iterations of training. For example, for T-shirts, after one training iteration, the coefficients take the shape of a T-shirt. It makes intuitive sense that the dot product of this with an image of a shirt will result in a high value. However, with more training iterations, this shape fades away until the coefficients look like unintepretable noise. One might suspect the model is simply overfitting, but validation accuracy does not start decreasing until after iteration 200, at which point the T-shirt shape is indistinguishable, and even then the overfitting is slight after that point.

What is the explanation for this phenomenon? Has the effect been discussed in the literature (does it have a name)?

The coefficients after a large number of iterations remind me of the coefficients that are typically learned by neural networks; is there a connection there?

There are two phenomena happening here:

1. This model learns typical features first before learning more particular features.
2. Overfitting manifests as noise. (Starting around 100 iterations)

The weight images become “less explainable” at first because they start to include less typical features. The weight for each pixel is initialized such that all the classes are equally likely. As a result, on the first iteration, you have all the training images of the correct class superimposed and all the images of the incorrect training classes subtracted. The result in this case looks like a typical example of the class. Look at the trousers for example. It looks like an average of all the trousers because that’s actually what it is! [1] (Ignoring the contribution of non-trousers examples) The problem is this does a poor job of identifying many training examples, for example, shorts.

As the model is trained, the typical examples are soon predicted accurately, so they have less influence on the gradient of the cost function. Instead, the gradient of the cost function is dictated by examples that are harder to predict. So changes in the weight images will be due to less-common features. Unless you study the training set carefully, it would be hard to explain the pixel weights because they are training on less-typical features.

Starting at 100 iterations, you have overfitting which is evident from the falling validation accuracy and the increasing noise in the weight images. Without regularization, any pixel can have an arbitrarily large effect on the activation of some class. We know this is wrong, but the model doesn’t know unless we impose regularization.

[1] Footnote

To see that the first iteration results in an equal superimposition of all the images on the weights, check how $$\theta_j$$, the weight for pixel j, depends on the value of pixel j $$x_j$$ after the first iteration:

$$\theta_j := \theta_j - \alpha \frac{\partial J(\theta)}{\partial \theta_j}$$

$$\alpha$$ is the learning rate for gradient descent, and the partial derivative $$\partial J(\theta)/\partial \theta_j$$ dictates how weight $$\theta_j$$ changes.

$$J: \mathbb{R}^n \to \mathbb{R}$$ is the cost of the training data given the parameters in column vector $$\theta$$. In the case of logistic regression without regularization we use the negative log-likelihood. This results in the partial derivative:

$$\frac{\partial J(\theta)}{\partial \theta_j} = \sum_{i\in\text{training data}} \left[\text{sigmoid}(\theta^T x^{(i)} - y^{(i)}) \right] x^{(i)}_j$$

On the first iteration, $$\text{sigmoid}(\theta^T x^T{(i)}) = 0.5$$, and since all $$y^{(i)}$$ must either equal 1 or 0 for positive or negative examples, respectively, the partial derivative for every pixel is either $$-0.5 x_j^{(i)}$$ or $$0.5x_j^{(i)}$$, so that every image either adds or subtracts itself from the weights equally on the first iteration.

$$\theta_{\text{iteration 1}} = 0.5 \alpha \left( \sum_{i \in \text{positive examples}} x^{(i)} - \sum_{i \in \text{negative examples}} x^{(i)} \right)$$

This shows that on the first iteration, every example image has equal influence on the weight image in either the positive or negative direction. After the first iteration, the likelihood for some examples will be closer to the truth, and those examples will exert less influence on the derivative.